#315737 - 01/11/2008 06:24
Voting Machines Redux
|
addict
Registered: 25/06/2002
Posts: 456
|
Haven't heard the usual furor about voting machines this year. Just the unfortunately now-standard commotion about "you're keeping our people away from the polls" and "but you're registering non-existent people".
But voting machines are still broken and dangerous and dangerously broken if I recall correctly.
Has the public just decided not to worry about this any more until some hacker makes 12 states vote 100% for Frank Zappa and the Hockey Mothers of Convention?
Anyone? Anyone? Wallach?
|
Top
|
|
|
|
#315739 - 01/11/2008 08:40
Re: Voting Machines Redux
[Re: music]
|
veteran
Registered: 01/10/2001
Posts: 1307
Loc: Amsterdam, The Netherlands
|
Well, Robert F. Kennedy is concerned, but I am surprised that there really hasn't been more outrage. But OK, I am partial - Harri Hursti used to work with/for me back in the old EUnet days...
|
Top
|
|
|
|
#315741 - 01/11/2008 09:13
Re: Voting Machines Redux
[Re: julf]
|
pooh-bah
Registered: 14/01/2002
Posts: 2489
|
Does it matter this time around? The way the media in the UK portrays it, Obama is the only candidate anyway.
|
Top
|
|
|
|
#315746 - 01/11/2008 11:58
Re: Voting Machines Redux
[Re: CrackersMcCheese]
|
addict
Registered: 14/01/2002
Posts: 510
Loc: NY
|
Does it matter this time around? The way the media in the UK portrays it, Obama is the only candidate anyway. It matters. If 3 votes in one state where one candidate is winning by a landslide, are counted wrong on purpose, it matters. And no, Obama isn't the only candidate at all, plenty of people are voting for McCain, it is close in several states (in my opinion, close is less than 10% apart. I don't quite trust the accuracy of polls.)
Edited by Heather (01/11/2008 12:12)
_________________________
Heather
"I distrust those people who know so well what God wants them to do because I notice it always coincides with their own desires." -Susan B Anthony
|
Top
|
|
|
|
#315747 - 01/11/2008 12:10
Re: Voting Machines Redux
[Re: CrackersMcCheese]
|
veteran
Registered: 01/10/2001
Posts: 1307
Loc: Amsterdam, The Netherlands
|
Does it matter this time around? The way the media in the UK portrays it, Obama is the only candidate anyway. He might be - but I am all too afraid Mccain might "win" anyway. Somehow. Magically. Unfortunately I guess it is too late to get independent election monitors in.
|
Top
|
|
|
|
#315748 - 01/11/2008 14:16
Re: Voting Machines Redux
[Re: CrackersMcCheese]
|
carpal tunnel
Registered: 25/12/2000
Posts: 16706
Loc: Raleigh, NC US
|
The way the media in the UK portrays it, Obama is the only candidate anyway. Oh, if only that were the case. The way US Presidential elections work, each state votes and then all of that state's votes go to the winner of the state. Right now, the polls have Obama with a significant lead taking that into consideration, but many of the state elections are very close. A small deviation from poll numbers could have a very significant effect. For a quick overview, take a look at Pollster.com or FiveThirtyEight.com.
_________________________
Bitt Faulk
|
Top
|
|
|
|
#315754 - 01/11/2008 20:22
Re: Voting Machines Redux
[Re: wfaulk]
|
pooh-bah
Registered: 14/01/2002
Posts: 2489
|
Interesting links, thanks.
|
Top
|
|
|
|
#315756 - 01/11/2008 21:54
Re: Voting Machines Redux
[Re: julf]
|
addict
Registered: 25/06/2002
Posts: 456
|
Or at least he was when that article was written a little over 2 years ago. but I am surprised that there really hasn't been more outrage. Agreed! With all the interest in this election, why hasn't this topic surfaced again? Also, thanks for the Hursti_Hack link! Interesting.
|
Top
|
|
|
|
#315757 - 01/11/2008 22:07
Re: Voting Machines Redux
[Re: music]
|
carpal tunnel
Registered: 29/08/2000
Posts: 14494
Loc: Canada
|
With all the interest in this election, why hasn't this topic surfaced again? Mmmm.. letmesee.. short attention span of USA media, perhaps?
|
Top
|
|
|
|
#315758 - 01/11/2008 23:35
Re: Voting Machines Redux
[Re: music]
|
carpal tunnel
Registered: 08/07/1999
Posts: 5549
Loc: Ajijic, Mexico
|
To steal a march on Jonathan Swift, I have A Modest Proposal. Why couldn't we have a system where people could vote on-line if they so chose? I've given this some thought, and of course the biggest problem boils down to identification of the voter, how to prevent fraud through people casting multiple ballots or ineligible people voting. I think the problem might be solved with biometrics. My wife's laptop computer has a little doohickey on it that scans her fingerprint to determine if she is authorized to log on. What if there were some sort of independent, non-partisan election clearing house whose duty was to tally internet votes. Those people who wanted to vote internet would have their index fingerprint on file. Those too paranoid to trust someone to keep one of their fingerprints would be relegated to voting the normal way, with paper ballots or voting machines. Now, it is unlikely that computers at the internet voting site would be able to look at a fingerprint and instantly compare it to 20 million fingerprints on file, so there would have to be a system of matching a single fingerprint to a known sample. This could be done by issuing a unique password to each person who registered to vote by internet. The voter would log onto the internet voting site, send his password and his fingerprint, and if the two corresponded he would be allowed to vote. The vote tally would be kept in a database with a simple structure that would be accessible to the voter after the polls closed. He could access (but not change) his ballot after the election and be certain that no fraud was perpetrated on it. In the same way, the entire database would be made accessible to auditors to make sure that the final tallys matched the data. Enough individual voters would check their ballots after the election that any widespread cheating would be detected immediately. Votes would be entered simultaneously into widely separated identical databases monitored by independent groups. Maybe the Democrats would have one, the Republicans another, one for the League of Women Voters, one for National Public Radio, you get the idea. This would make things pretty much tamper-proof, as each database could be compared to the others to be sure they all matched. Registering to vote by internet would be done the same way voter registration is handled now -- proof of citizenship, ID and residence, plus providing a fingerprint. The registrant would be on the internet voter registration list, and removed from the "at the polls" list. One concern to be addressed would be privacy. Since the ballots would be accessible through the voter's password, and the voting website would have to have a list of passwords to verify that the voter was eligible to participate, it would theoretically be possible to match individuals to their ballots. To prevent this, the passwords would not be kept with the voter registration information, but just sent on to the vote talley site along with the corresponding fingerprint without any other identification. There are other problems, like what is the procedure if a voter pretends to forget his password? How would you purge his old password and fingerprint from the voting website? I guess the big question is, could this idea actually work? I am not suggesting that everybody be required to vote this way, it would just open up another avenue of voting that would greatly enhance voter turnout. You could extend the voting period for internet voting without incurring additional expense -- let the internet voters begin voting a week before the polls closed. Even people without computers could vote by internet, internet access is available at every library in the country, and probably most of the Starbucks as well. When I get to be King of the World, that's how the elections will be run. Except for the one that elects the King, that is. tanstaafl.
_________________________
"There Ain't No Such Thing As A Free Lunch"
|
Top
|
|
|
|
#315761 - 02/11/2008 00:41
Re: Voting Machines Redux
[Re: tanstaafl.]
|
carpal tunnel
Registered: 29/08/2000
Posts: 14494
Loc: Canada
|
The biggest (and most obvious) problem with such a scheme, is, most of the voters would probably be using Microsoft software, along with many layers of the associated malware.
It would be incredibly easy to hijack all of their votes, just as nearly all of their machines today are used to remail spam.
-ml
|
Top
|
|
|
|
#315762 - 02/11/2008 00:59
Re: Voting Machines Redux
[Re: mlord]
|
carpal tunnel
Registered: 08/07/1999
Posts: 5549
Loc: Ajijic, Mexico
|
It would be incredibly easy to hijack all of their votes No, it wouldn't. The only way a ballot could be added to the database would be with a password and the registered voter's fingerprint. Any hijacking would be immediately visible. If thousands of voters' ballots were missing or changed in the database when the voters checked on them, a big fuss would be made immediately. It wouldn't matter what software was used to place the ballot into the secure, audited, linux-OS database. Once the ballot got in, it would be secure and could be be immediately confirmed by the voter as being correct. tanstaafl.
_________________________
"There Ain't No Such Thing As A Free Lunch"
|
Top
|
|
|
|
#315764 - 02/11/2008 02:00
Re: Voting Machines Redux
[Re: tanstaafl.]
|
carpal tunnel
Registered: 06/10/1999
Posts: 2591
Loc: Seattle, WA, U.S.A.
|
It would be incredibly easy to hijack all of their votes No, it wouldn't. The only way a ballot could be added to the database would be with a password and the registered voter's fingerprint. Any hijacking would be immediately visible. If thousands of voters' ballots were missing or changed in the database when the voters checked on them, a big fuss would be made immediately. It wouldn't matter what software was used to place the ballot into the secure, audited, linux-OS database. Once the ballot got in, it would be secure and could be be immediately confirmed by the voter as being correct. tanstaafl. When it comes to IT and systems, I sometimes use something I call the "alien test". If a flying saucer landed and the little green gent got out and asked "Hey, what are you doing?" could we explain what we were doing and why in a way that the alien could appreciate? Or would he think we were nuts and destroy us with his death ray because we made him irritable? I find this test very useful when it comes to issues of complexity -- is what we are doing as simple as possible and only as complex as is necessary? If what we are doing is very complex, can we tell the green dude why? I'm probably a bit of a wet blanket, but I think that any online scheme involving multi-factor authentication would have a lot of areas of exposure/risk -- hijacked computers/browsers, stolen thumbs, diversion of identified voting results to the King ("Off with their thumbs!"). While I would like something better, the simple unassociated nature of putting an anonymous ballot in a bubble reader works for me -- and absentee ballots and motor-voter stuff that just makes it easier to vote without much more complexity and that are generally usable by most/all potential voters. I have to say that the notion of somebody sitting in their 43rd floor penthouse and sliding their thumb over a reader while the lumpen underclass struggles below with a broken pencil, well, it kind of seems like something out of a Heinlein novel!
_________________________
Jim
'Tis the exceptional fellow who lies awake at night thinking of his successes.
|
Top
|
|
|
|
#315774 - 02/11/2008 07:59
Re: Voting Machines Redux
[Re: tanstaafl.]
|
veteran
Registered: 01/10/2001
Posts: 1307
Loc: Amsterdam, The Netherlands
|
|
Top
|
|
|
|
#315778 - 02/11/2008 12:09
Re: Voting Machines Redux
[Re: tanstaafl.]
|
carpal tunnel
Registered: 29/08/2000
Posts: 14494
Loc: Canada
|
It would be incredibly easy to hijack all of their votes No, it wouldn't. The only way a ballot could be added to the database would be with a password and the registered voter's fingerprint. All of which passes through the malware, which can hack it at will. Any hijacking would be immediately visible. Not from the same PC it was originally intercepted by.. There's a lot to be said for paper ballots. Cheers
|
Top
|
|
|
|
#315782 - 02/11/2008 15:35
Re: Voting Machines Redux
[Re: tanstaafl.]
|
carpal tunnel
Registered: 25/12/2000
Posts: 16706
Loc: Raleigh, NC US
|
The biggest problem is the disenfranchisement of those without access to the internet. Yes, I know that voting in person would still exist, but you're lowering the bar for the wealthy, and making a distinction between classes is bad. You're going to make it so that the bar to voting for the wealthy is taking five minutes whenever you want, whereas the bar for voting when you're not is standing in line for half a day during business hours.
The other problem is the ability to associate votes with an individual. The current notion is that while votes are initially counted electronically, they are counted off of a piece of human-readable paper that the voter can easily read and that can be easily read by other humans in case of a recount. The reason you want people to be able to see their vote is so that they can have that same feeling of trust that the votes that they cast are the votes they intended. However, it also allows people's votes to be verified by third parties, thus creating an avenue for the purchasing of votes. Right now, if someone pays someone else to vote, there is no way to verify what his vote was. But under your system, the vote purchaser could require that the voter show his vote confirmation before getting paid.
Note that the problem you're trying to solve is the same one being brought up with the advent of electronic voting machines. The obvious solution is for the electronic voting machine to produce a human- and machine-readable paper ballot that is then tallied by a different system. You still get the benefits of a simpler UI, but you gain the ability for the voter to verify his vote before it is tallied.
_________________________
Bitt Faulk
|
Top
|
|
|
|
#315783 - 02/11/2008 15:53
Re: Voting Machines Redux
[Re: wfaulk]
|
carpal tunnel
Registered: 18/01/2000
Posts: 5683
Loc: London, UK
|
You still get the benefits of a simpler UI I've never quite understood why the "UI" of voting was the source of such problems. Surely it's just a sheet of paper with a list of names, each with a big box next to it? You put a big X in the big box corresponding to the name you like the sound of. Job done. You can then use Optical Mark Recognition to figure out which big box the big X was put in, and therefore which name most people like the sound of.
_________________________
-- roger
|
Top
|
|
|
|
#315793 - 02/11/2008 17:58
Re: Voting Machines Redux
[Re: Roger]
|
carpal tunnel
Registered: 30/04/2000
Posts: 3810
|
Wow, let me see if I can address some of the questions here. This topic is a big part of what I do for a living... Why not vote on the Internet? mlord sums up one significant objection: evil malware on the voting terminal could show one thing while doing another. A second objection is that somebody could watch over you shoulder while you vote, thus voiding your privacy and enabling voter coercion / bribery. Estonia tried to address this by letting you vote multiple times, with only the last one counting. That means that there's a big database somewhere that has your name next to your vote. Not reassuring. Also, you authenticate via your national ID card (a smartcard thing). Somebody could watch you vote then "borrow" your card until the election is over. Sophisticated "end-to-end" cryptographic techniques can work around the malware platform issue, but cannot do anything about the coercion issue. See, for example, Helios. (We're using similar crypto machinery in our VoteBox voting system, which appears more like a "standard" electronic voting machine.) Is the media following this issue? My phone seems to be ringing off the hook. I was on NPR Science Friday last week. I'll be on several other radio stations this week. The press is definitely covering the issue. Of course, it's not the #1 problem these days. So what about this UI problem? Surely it can't be that hard to do? Actually, it can, because we're not dealing with "normal" computer users. We're dealing with every registered voter, which means older people who don't use computers and illiterate people and non-native speakers, and on and on. The current, most likely culprit, particularly notably in West Virginia for some reason, is typically misnamed "vote flipping", and is really the result of touch-screen calibration issues. If you hit what you think is the proper area for your candidate and it selects the one above, you (computer geek, reader of this board) will simply aim lower and try again, knowing that you're compensating for a bogus calibration. A non-trivial percentage of voters won't know to do that.
|
Top
|
|
|
|
#315794 - 02/11/2008 18:01
Re: Voting Machines Redux
[Re: DWallach]
|
carpal tunnel
Registered: 30/04/2000
Posts: 3810
|
And one other thing, in response to wfaulk: surely the answer must be to attach paper printout gadgets to electronic voting machines, right?
Sadly, no. I was one of the big proponents of this idea until I saw the results of human subject experiments. The punch line: 63% of test voters failed to notice deliberately inserted errors on the summary screen. (And the test subjects just loved using our evil, lying machine!)
This is why I'm increasingly favoring the crypto solutions, which let you catch a machine as it's cheating.
|
Top
|
|
|
|
#315798 - 03/11/2008 01:43
Re: Voting Machines Redux
[Re: Roger]
|
carpal tunnel
Registered: 25/12/2000
Posts: 16706
Loc: Raleigh, NC US
|
I've never quite understood why the "UI" of voting was the source of such problems. Surely it's just a sheet of paper with a list of names, each with a big box next to it? Do you mean that this is the way you think voting (in the US) does work, or that it's the obvious solution? This is the contentious ballot from Florida in 2000. Seems pretty straightforward, right? And the NC (where I live) ballot(pdf) is being commented on as this year's poor UI ballot because of the presidential vote being distinct from the straight-party vote. Neither of those really seem very difficult, do they? But people are idiots.
_________________________
Bitt Faulk
|
Top
|
|
|
|
#315799 - 03/11/2008 02:25
Re: Voting Machines Redux
[Re: wfaulk]
|
carpal tunnel
Registered: 30/04/2000
Posts: 3810
|
It's not exactly that people are idiots. It's more that some tasks are intuitive, some tasks are trained, and other tasks are counter-intuitive or counter to prior training.
Canonical example: how many doors have you seen with a "push" or "pull" sign on them? If you have to explain it, you've already failed. If a door is a "push" door, then you shouldn't have a pull handle on it!
Car nerd example: recent BMWs have been dorking with the turn signal indicator. Rather than what we all know and love, where the turn signal physically locks up/down until you turn the wheel, they've been doing it all in software. Guess what? People will get it wrong because it's just weird and different.
Now, let's take North Carolina's unique rule that "straight ticket" voting doesn't apply to the presidential race. This is giving both parties fits, since they now feel the need to generate instructions for voters ("vote straight ticket then vote for our guy for president"). Those same instructions, once they cross the border to a different state with different rules, are now a recipe for getting voters to deselect their presidential candidate preference.
To wrap your brain around the extent of the problem, take off your computer wonk hat and put yourself in the shoes of somebody who's never touched a computer beforehand. How are you supposed to figure out whether you've got it right? It's not so simple any more.
|
Top
|
|
|
|
#315800 - 03/11/2008 04:29
Re: Voting Machines Redux
[Re: DWallach]
|
carpal tunnel
Registered: 08/07/1999
Posts: 5549
Loc: Ajijic, Mexico
|
Neither of those really seem very difficult, do they? Oh, sure, perfectly straighforward. No chance of confusion here... a. A Straight Party vote is a vote for all candidates of that party in partisan offices.
b. You may select a Straight Party AND ALSO vote for a candidate of a different party
c. In any multi-seat race, a Straight Party vote is a vote for ALL candidates of that party.tanstaafl.
_________________________
"There Ain't No Such Thing As A Free Lunch"
|
Top
|
|
|
|
#315801 - 03/11/2008 06:16
Re: Voting Machines Redux
[Re: DWallach]
|
addict
Registered: 25/06/2002
Posts: 456
|
Is the media following this issue? My phone seems to be ringing off the hook. I was on NPR Science Friday last week. I'll be on several other radio stations this week. The press is definitely covering the issue. Of course, it's not the #1 problem these days. Thanks, Dan. I'll look for and grab the Science Friday podcast. I'm glad the media are covering it. I guess I haven't seen anything about it this year (besides the bogus "vote flipper" thing), though I've heard more than enough about pigs, lipstick, acorns, plumbers, expensive wardrobes, crazy preachers, counting your residences, wearing traditional costumes, how many countries are visible from which point, how much both candidates love Jesus and how much they'll both cut my taxes, not cut out a single one of their campaign promises, and still somehow magically fund this surprise trillion dollar extra outlay. Yet somehow I'm supposed to care more about all of that silly nonsense than the fact that our vote tallying process is less robust than it was 20 years ago. ...anyway... It seems to me that voting-machine-related fraud is one of the biggest threats to the "sanctity" of our democracy. (Yes, idiocy of the electorate is a bigger long-term problem, but in the meantime, let's make sure that every idiot's vote is properly counted, and then try to convince people to make rational decisions and that they ought to demand more from the candidates than soundbites and detail-free "debates".) I.e., - the government should make good on its end of the bargain to properly record what the people ask for, and
- the people should try to make good on their end of the bargain and vote on more than just color, party, "what's in it for me?", propaganda, emotion, or one single pet issue.
This is especially true for senate and house elections.
|
Top
|
|
|
|
#315802 - 03/11/2008 06:54
Re: Voting Machines Redux
[Re: wfaulk]
|
carpal tunnel
Registered: 10/06/1999
Posts: 5916
Loc: Wivenhoe, Essex, UK
|
I certainly find that Florida one very confusing, but then it is unlike any ballot paper I have seen in the UK, so maybe if I was used to US ballot papers it would have been clearer. That staggered layout seems guaranteed to cause confusion.
I keep hearing that people are standing in line for hours in the US to vote. Is this really true ? Is this normal ? Does this happen in all areas or maybe just in poor areas ?
I can't remember ever having to wait more than 20 minutes to vote in the UK. If you had to wait for hours I imagine the UK turnout would be even more appallingly low than it already is.
_________________________
Remind me to change my signature to something more interesting someday
|
Top
|
|
|
|
#315803 - 03/11/2008 07:06
Re: Voting Machines Redux
[Re: wfaulk]
|
carpal tunnel
Registered: 18/01/2000
Posts: 5683
Loc: London, UK
|
This is the contentious ballot from Florida in 2000. Seems pretty straightforward, right? No. It doesn't. A ballot paper should have a column of names on the left, and a column of big boxes on the right. The problem with that one is the fact that the names alternate left and right, and that the "boxes" aren't big, and that they're not boxes. Your second one could be solved by having a separate piece of paper for each post. It worked fine for us in London earlier this year. Admittedly, we're still counting ours by hand, but "putting a big X in a big box" would work fine with OMR.
_________________________
-- roger
|
Top
|
|
|
|
#315804 - 03/11/2008 07:27
Re: Voting Machines Redux
[Re: Roger]
|
pooh-bah
Registered: 12/01/2002
Posts: 2009
Loc: Brisbane, Australia
|
My god that is a badly designed "interface" and I'd consider myself to have a pretty good eye... The arrows don't even line up with the exact centre of the dot. No wonder people got confused.
_________________________
Christian #40104192 120Gb (no longer in my E36 M3, won't fit the E46 M3)
|
Top
|
|
|
|
#315807 - 03/11/2008 11:30
Re: Voting Machines Redux
[Re: wfaulk]
|
carpal tunnel
Registered: 29/08/2000
Posts: 14494
Loc: Canada
|
Wow.. that Florida ballot is almost criminal! No wonder the Republicans won that state -- just gotta be at the top of that particular ballot and you're guaranteed more votes!
And the NC ballot.. mmm.. too much reading required for a crowded voting room. But even so, it actually encourages blind voting along strict party lines (except for the pres/veep), which only serves to further polarize politics and the country.
Dumb, dumb, dumber.
Oh well.
|
Top
|
|
|
|
#315808 - 03/11/2008 13:48
Re: Voting Machines Redux
[Re: andy]
|
carpal tunnel
Registered: 25/12/2000
Posts: 16706
Loc: Raleigh, NC US
|
I keep hearing that people are standing in line for hours in the US to vote. Is this really true ? Is this normal ? Does this happen in all areas or maybe just in poor areas ? First, this is "early voting", which has only been in existence in the US for about 10 years or so. Given that a large number of voters only vote every 4 years, that's not a lot of iterations. Before then, everybody voted on a single day. The difference is that there are far fewer early voting places than normal voting places. For example, in my county there were about a dozen early voting places, but there will be about 200 polls open tomorrow. But the early voting places were available for weeks. Meaning you could find a good day to spend a while voting. I'd say that, generally speaking, people wait no more than an hour in an average election, and probably far less time than that in the majority of cases. However, this is no average election. In NC in 2004 (the last presidential election), about 3.5 million people voted. This time, it's expected to be around 4.6 million. 2.6 million have already voted. Yes. People are standing in line for hours. And it's because they have unusually strong feelings about this election. At least part of that is due to Bush's approval ratings being in the ranks of the lowest scores ever received by a US president over the past few weeks. I have no idea what the actual election day lines will look like tomorrow. But I'm going to find out.
_________________________
Bitt Faulk
|
Top
|
|
|
|
#315809 - 03/11/2008 13:50
Re: Voting Machines Redux
[Re: mlord]
|
carpal tunnel
Registered: 25/12/2000
Posts: 16706
Loc: Raleigh, NC US
|
it actually encourages blind voting along strict party lines Yeah. I'm all in favor of getting rid of straight-ticket voting. But it's not like NC is the only state that does that. I'd be surprised if there were many states that didn't.
_________________________
Bitt Faulk
|
Top
|
|
|
|
#315810 - 03/11/2008 14:12
Re: Voting Machines Redux
[Re: wfaulk]
|
addict
Registered: 25/06/2002
Posts: 456
|
I have no idea what the actual election day lines will look like tomorrow. But I'm going to find out. I, too, find these long lines surprising. The last 2 or 3 times I've gone to vote, there have been 4-6 voting stations and 1-2 voters at them. In fact, in the past decade, my maximum voting wait time has probably been 2 minutes. Maybe that's because California has elections all the time and people don't bother to show up due to voter fatigue, or maybe it is because everybody in my region mails their ballot in far in advance. Or maybe it is because California's vote is always so lopsided that people don't feel it necessary to show up. In any case, I'll be very curious to see if there are any lines at all when I go to vote tomorrow. Perhaps there will be because we have several hotly contested propositions this time. (Gay marriage, and a couple of alternative energy propositions.)
|
Top
|
|
|
|
|
|