I emailed them. I live I'm SF. This is the sake, solved, problem we have in the machine tool world (milling machines, lathes, etc). Similar to old keyboards, you can just buy and use a floppy emulator.
They emailed me back, they said that the floppy thing makes a good headline but is really just the tip of the iceberg. It's really the whole system that's like this at every layer, it needs replacing they say.
This was my thought. Floppy emulators are a dime a dozen thanks to the retrocomputing community. Emulators for the custom boards full of discrete logic, opaque ROM chips, PLAs, and so on from long dead companies are going to be a much bigger challenge.
If they had good schematics for all of the parts it might be possible to keep the the system running for a long time with a couple of smart EEs who are comfortable with the scope and soldering iron, but eventually they're going to run out of some obscure part and be up a creek.
Or maybe they could replace entire boards with home designed versions condense all of the old logic down to one chip and a handful of support components and start in-place upgrading without a total system revamp. Still an expensive process, and one that requires some hard to find engineers on staff, but theoretically spreads out the upgrade process over many years. It also loses out on functional improvement opportunities while your system is made up of a hodgepodge of old and new hardware.
SelTrac was designed to use commercial off the shelf (COTS) parts so there shouldn't be much in terms of needing custom chips, etc. The problem is that Alcatel (now Thales) basically doesn't support any of this (and hasn't for a long time).
The computers in question are, I believe, just thin clients.
RE "....Still an expensive process, ....."
Actually probably NOT , if all the $$ saved over the last 25 years , and also considering the many millions to get a completely new replacement system....
The expense will be in "certifying" the machines for use with passenger traffic. Nobody wants personal liability, so we wrap ourselves in knots trying to pass responsibility on to someone, anybody, else.
A once a generation refresh seems okay in my book. The Breda trains that came in 1996 were being phased out about twenty years into their tenure, most generally seem to think that is about reasonable a time to do something like that. I don’t see software related regimes as inherently that different.
Since I experienced 5.25 inch floppy disk era...and even the occasional bernoulli disk...we could simply say: the system had its run, replacement is reasonable. A lot of stuff had changed since then, and not just in storage media.
That ancient code is the crystalization of the knowledge of how to run, at the most basic level, all of the MTA. Emulating hardware is a good way to extend it's life while the decade long task of replacing it is underway.
Until you have directly used "legacy" machine that are mission critical, and understand how even a tiny error could cause the failure of a business and all the jobs that go with it, you can't avoid underestimating the true scale and scope of the problem.
Here in the Washington DC area, our metro still has two series of rolling stock that date back to 1982 and 1987 (2k and 3k series). At the earliest they won't start retiring them in 2024 and 2025 at the earliest...
I think they're just saying parts should be periodically replaced across the board on that timeframe, and using the fact that the trains are being swapped out as an example. The Breda trains are still running some percentage of the fleet however. I rode in one today.
That’s right. System refreshes are a fact of life, especially if one is parsimonious in using what works until obsolescence. There are downsides of doing things that way, but I don’t see it inherently more defective than other approaches, such as some kind of continuous refresh that has a conceit of keeping a system “modern” in perpetuity.
But twenty years for a train seems very low. My daily driver is 12 years old, I thrash it every morning and never gives me problems. I live in the snowbelt.
By comparison a train is vastly over-engineered and has an electric drive train. Sure stuff will need replacing but a train should last 40 years at least
I wouldn't call myself as particular on train maintenance...but consider that these trains are run more or less continuously. You thrash your car in seasonally tough conditions every morning, but these drive all day and most of the night. So, I'm not sure if the comparison is apt on these grounds.
A cursory search suggests that quite a few manufacturers design for light rail car lifetimes of 25-30 years, not forty. These tend to be of European origin, which tracks San Francisco switching from an Italian to a German vendor. I don't see evidence that it's a common practice to significantly extend the tenure of those devices there nor here.
Funnily enough, I see no problems with the Breda trains personally as a passenger. But once they've shaken out the major bugs, the Siemens train reliability is anticipated to be triple or more. It may not make sense to design a traincar for a fifty year term.
Transit vehicles purchased with federal funds are required to meet a minimum service life (see below). For the Muni Metro you're looking at a minimum expected life of 31 years.
That's an average service life from what I can tell, not a minimum (a minimum average?). But yes, it does trend a tad higher than what my cursory search indicated (25-30, feds want 31)
Looks like this has been the subject of some industry wandering in the last generation:
Some were planned for 30-35 years, some for 25 to 30, some models have aged so gracefully they may be refurbished, yet others may have been engineered with roughly the correct obsolescence window in mind as engineering has advanced in the last generation, even in the dimensions of making some things simpler and more reliable (one of the billed advantages of some of the Siemens train mechanisms).
FTA has set a default ULB as the expected service years for each vehicle
class in the table below. … In cases where the submitted ULB differs
significantly from the default value, agencies may be prompted to submit
justification.
The first generation of modern streetcars (the Boeing-Vertol) barely made it 24 years and that was scandalous. The next generation (Breda) were so bad that their manufacturer was banned from city contracts. Even so the Boeing cars were retired post haste as Muni took deliveries of the Breda cars. The last of the Boeings were decommissioned in 2001, and the Bredas look like they'll be around the full 31 years at least.
Basically if you're buying things that don't last 30 years, the feds are going to be very reluctant to give you money.
The real scope of the project is a mind-boggling $600 million to track the light rail trains inside and outside the tunnels and to control the trains inside the tunnels. Divided by the 250 light rail vehicles (apparently does not include historic streetcars), it works out to $2.6 million per car. Plus $36M for consulting (2023-11-07 board presentation from the documents that I linked earlier https://www.sfmta.com/sites/default/files/reports-and-docume...).
The PC and the floppy disk should be running as a virtual machine. ( The Vm software and host OS and the hardware would all be current supported versions of course )
That doesn't result in bigger budgets. If the goal of a manager is to increase their budget and hence scope of responsibility, any solution which doesn't result in a bigger budget isn't really a solution. Same reason it's going to cost a hundred million, yet a single indie game dev can write a train scheduling simulator by themselves.
the state-of-good-repair grift will continue until the morale improves! SelTrac is fine. and if it ever actually breaks (dubious) they can fall back to block signaling which comfortably handles more trains than muni is capable of running. the MBTA does just fine with it on the green line, and when the MBTA is making you look incompetent you've got to question where it all went wrong.
this is the agency that brought you the central fucking subway, a $300M sewer project masquerading as some red paint on van ness, that picked the already-mostly-grade-separated M Ocean over the N Judah to subway-ify and that is incapable of flicking the traffic preemption switch on the rest of the T to the on position without a decades long pedestrian detection LIDAR project for the unique in the world needs of san francisco.
It sounds like they let this problem fester until it has reached this existential end. They clamor for new technology, which they get, then they use the most risk adverse management strategy and never upgrade or change it, until it reaches this problem state.
They either need significant third party help deploying and managing this system, or they should go back a few generations of technology and use the simplest possible system that meets their needs. Pen and paper should be considered if it can be made more efficient.
I guess the people who work on public transit aren't interested in having the best public transit system available. They're only interested in keeping it running for as long as possible with zero changes or responsibility.
This is why I'm strongly doubtful on public transportation in the US. Our bureaucracy can't handle it.
> I guess the people who work on public transit aren't interested in having the best public transit system available. They're only interested in keeping it running for as long as possible with zero changes or responsibility.
Let me guess, you also think public servants are lazy, overpaid and unskilled? Comments like this belie how many people form opinions of public agencies without having any real experience with government.
The problem is the public and the politicians - transit employees largely do the best they can with the resources and constraints they are given. In order to succeed transit needs municipalities and states to: pay more taxes; accept that transit won't generate a profit, but does generate non-monetary public value; and gather their resolve to deal with NIMBYs.
>Let me guess, you also think public servants are lazy, overpaid and unskilled? Comments like this belie how many people form opinions of public agencies without having any real experience with government.
I've worked in the public sector for several years, and there are definitely a decently-sized group of people in many different positions that are riding out their time until retirement trying to fly under the radar and do as little as possible. Not to say there aren't people that aren't competent, but as a competent person you have to work around the people that don't care and often actively don't want to see more work come their way to get anything done. It becomes tiring and eventually the private sector is too enticing and the competent people leave.
Transit employees are not the same people as, nor have the same motivations as the politicians who manage transport agencies.
Of course the employees work hard. It’s the vocation they’ve chosen. I’m always amazed by the diligence of TFL and rail employees in the U.K., for instance.
No, the guiding principle for the folks at the top is “so, how can I make money for myself out of this before I fail upwards?”.
I’d wager the SF situation is just the same old looting by the Very Important People who run the show, leaving employees and actual managers with a shoestring and two peanuts to make it all work, while the head honchos bathe in gravy.
> No, the guiding principle for the folks at the top is “so, how can I make money for myself out of this before I fail upwards?”.
Who are "the folks at the top"? Transit agency executives in California make less than software engineers (though if you put in your time, you get a nice pension to make up for it) - how are these people getting rich? If you mean the politicians, sure, I've already said they're half the problem.
There is always stable "Go work as consultant at company that does a ton of business with former agency. This position is in no way related to any agency work they did. :wink:"
> Of course the employees work hard. It’s the vocation they’ve chosen. I’m always amazed by the diligence of TFL and rail employees in the U.K., for instance.
We are talking about American transit workers. I haven’t been to London, but I’ve been amazed by the diligence and efficiency of Japanese transit workers. But they bear zero resemblance to the folks running the transit system in DC, where I live.
As I learn more about how this city operates, living here longer, I keep finding myself truly amazed at how corrupt this city is. Even if you try, you can't avoid it. It's truly everywhere.
You underestimate. The more places you go, the more you see, the more you do, the more you realise it is truly EVERYWHERE.
I honestly find it incredibly disheartening how rampant and underreported corruption and nepotism are in practically every sector, industry, everything, everywhere, always.
The corrupt are a minority - but they are global, and the impacts of their actions are damaging the entire human race in a very significant fashion, and always have.
> Let me guess, you also think public servants are lazy, overpaid and unskilled? Comments like this belie how many people form opinions of public agencies without having any real experience with government.
I think American public servants are lazy, overpaid, and unskilled, yes. You don’t need experience running the government to know that the government sucks at its job, just like Windows users could tell Windows 95 sucked without being programmers themselves.
US transit systems almost uniformly suck compared to those in other developed countries, while costing the public just as much money, and in many cases a lot more money: https://www.forbes.com/sites/stephensmith/2011/12/11/surpris.... (“ The Swiss Transit department made a presentation a year or two ago in San Francisco. As far as I could tell, the public transit subsidies for all of Switzerland are in the neighborhood of the subsidies for transit in the Bay Area…
He gives total [Swiss] transit subsidies at $1.5 billion.The 2009 MTC report gives Bay Area transit subsidies at $1.3 billion.”).
Your experience as a customer of a transit service is based on the aggregated output of public employees in general, not individuals. I’m sure there are hard working, efficient people at WMATA. But those individuals can’t single handedly keep the system from falling apart when their coworkers are trying to do as little as possible.
Once again, I will reiterate that right wing characterizations of public employees are very silly, not reflective of how most municipal agencies operate, and is intended to erode public institutions. Public employees are not inherently more lazy or less skilled than private sector employees. Their incentives and challenges are different. If the performance is subpar, look in the mirror to find the problem.
Its completely besides my point, but comparing sprawling American metros to compact Swiss cantons is not a useful exercise. If we want to redevelop American cities to be more transit friendly, I am on board!
I’m very left wing on transit. I rode the Amtrak from Baltimore to DC every day for two years, I commuted from the suburbs to various DC metro stations for years to commute into the city. The right wing characterization of transit proved true based on experience despite my strong desire for it not to be true. Don’t get mad at right wingers for simply pointing out what’s plain to see. There’s a reason Americans are overwhelmingly leaving the places that have good transit for places that have no transit—the American version that’s always slow, late, dirty, and doesn’t go where you’re going is just not an amenity that’s worth it for most people.
The problem is not funding or public policy or geography. The northeast corridor is densely populated; there is no reason in terms of geography why you shouldn’t be able to have efficient transit in Baltimore and DC. These systems are well funded and heavily subsidized by the federal government. The problem is the people, and that’s a problem you can’t fix. You cannot redevelop American cities into anything other than what they are so long as you have Americans running them. It’s a futile exercise.
> Let me guess, you also think public servants are lazy, overpaid and unskilled?
Do you not see this article as evidence of this fact? If you walked into any other business reliant on "floppy disks" just to take your money, would you praise their practices and efficiency?
> Comments like this belie how many people form opinions of public agencies without having any real experience with government.
I form opinions because I try to use their services and am disappointed and often disgusted. Do you really think "experience with public transport" is particularly hard to come by? That this is all too esoteric for the "common man" to opine on? Please...
> transit employees largely do the best they can with the resources and constraints they are given
Perhaps they should earn those resources competitively instead of being given them? Otherwise, this seems like a perennial chicken and the egg problem that swallows tax dollars with no worthwhile result.
I don't think the article is a reflection on efficacy of the public agency. If their budget for new software is $0, how exactly are they supposed to upgrade it? It's certainly possible that the agency has been utter shit for 25 years, but this article doesn't give the information to conclude that.
> Perhaps they should earn those resources competitively instead of being given them?
How? They're a public agency. There is no profit motive - if the public wants nicer stuff, they need to pay for it. That includes both the software costs and the higher salary costs to hire people capable of those types of projects.
Again, you come off as someone running their mouth with no actual experience interacting with public agencies. It's very frustrating to know how to fix a problem without being given the resources or permission to do so, which is often the case in transit. Add to that most transit agencies have a tiny budget for IT, and I'm not sure how you expect this to get solved without a big public push.
The problem is "lower my taxes" single issue voters. People vote for whomever lowers taxes and then complain about potholes. It's always the same people. GOVERNMENT IS INCOMPETENT BLAH BLAH. Yeah, cause you refuse to fund it and everything goes to shit, then is 10x more expensive to fix it compared to properly funding maintenance. You end up paying more in the end because these people can't see past the couple hundred dollars they saved on their property taxes and wonder why everything is turning to shit.
It’s weird that people have this notion about government and only government. There are things governments do well and things they do badly. You could say the exact same thing about private business: businesses go bankrupt every day, wasting time and resources. But no one argues that the very concept of business is “incompetent”.
If a business fails to perform well, its customers can choose another provider, and the business will have to improve, or else it will fail.
If a government fails, what recourse do its "customers" have? Wait a few years for the next election and hope that the truth about the government's poor performance wins out over the propaganda, and hope that the citizens vote in competent, honest officials, who might gradually make improvements, despite having to work through non-elected officials who are difficult to replace for incompetency? How often does that happen? And what if the officials in question are merely appointed by those who are elected, and thereby insulated from accountability by several levels of indirection? What recourse do the "customers" have then? Can they choose another "rail provider"?
The private sector has natural consequences for failure and a natural incentive to excel and be efficient (certain circumstances, such as monopolies, notwithstanding). The public sector is heavily insulated from consequences for failure and inefficiency.
That doesn't mean that everything should be privatized. It means that we shouldn't be surprised when a service provided by the public sector fails to perform well or efficiently, and we should carefully consider what the government ought to be responsible for. Also, we should make it easier to hold government officials accountable and replace them for incompetence. If someone wants the security of a public sector job, they ought to be held to the highest standards.
If a business fails to perform well, its customers can choose another
provider, and the business will have to improve, or else it will fail.
In this case, mayor Willie Brown (who predates the creation of SFMTA) steered the contracts for the trains and train control to Breda and Alcatel/Thales via Booz Allen Hamilton. Essentially he set up a few generations of sole source contacts for train control as you can see when the MTA was brining the Siemens cars and central subway.
Thing is, Alcatel did fail. Thales scooped them up. From a customer POV Alcatel/Thales failed too. They sold an end of life product that the MTA had to spend millions on just to add a new route (first part of the central subway). They've largely abandoned support of the inductive loop system.
Sure, the MTA could choose another vendor. If memory serves that's where they're stuck currently. Of course this brings out all the anti-government folks despite the main problem being a private vendor (Thales) and private consultants (BAH).
It reads like you are conflating newer with better.
It was more than a few years ago, but I remember the posts on Slashdot where tech people were wringing their hands over the fact that election offices were not adopting touchscreen ballots fast enough. Some posters were also promoting things like internet voting. How did that work out?
Sometimes it actually is better to move slowly instead of adopting something new just because it is new. With old equipment and older technology, we know the failure modes, we have the procedures in place for addressing them. Whether it's floppy disks, paper ballots, (or human beings,) older does not necessarily mean worse.
or they're not immune to the same urge all SW devs have to rewrite - it's a lot easier to start from scratch than (a) figure out how an existing system works, (b) figure out how to upgrade or modernize
My company provides new compute modules for another major global city's infrastructure that still relies on Intel CPU's from the early 80's. That is, we are building them brand new boards populated with chips that are over 40 years old.
They haven't shown any interest in updating the system. It works, they can get service, and get "new" replacements for things that go bad.
What they might not know though is that there is basically just one engineer we have (and probably the only one on Earth) who knows how to work on these things. He's getting old, and obviously none of the younger engineers really have an interest in learning ancient forgotten systems.
LOL:) this story seems to resurface every couple of years. My favorite part of the article:
"Jeffrey Tumlin: "It's a question of risk. The system is currently working just fine but we know that with each increasing year risk of data degradation on the floppy disks increases and that at some point there will be a catastrophic failure."
This seems to imply they have been using the exact same disk for the past 20 years (absurd), they have absolutely no idea what is written on the disk and how it can be safely backed up or restored. This would be a problem regardless of the medium used.
Although I hold the line at using paper tape there is nothing wrong with using floppies other than it seems antiquated. It certainly is reliable and cheap. Maybe the only thing that needs replacing is the people running the Muni.
Every sizable manufacturer of floppy disks has exited the market. There are no more (major) suppliers for the tech. They’ve likely been depending on a dwindling supply of functional disks; if at some point they find themselves without enough working disks to operate the system, they will indeed be screwed.
There’s also the chance that they take a disk that’s on the verge of failure, plug it into the system, and some corrupted commands get loaded into the system. That could easily result in a “catastrophic failure”.
Floppy disks are not reliable or cheap. They physically degrade over time, and at this point are nowhere near cheap for “new” disks.
You make a good point. However, even with the disks backed up to a better storage medium, detecting data loss on the floppy and getting a replacement written and deployed to the correct location might take a while. During that time the system may face unacceptable transit delays.
Wait, what? If this thing boots from a floppy, and that floppy has a checksum on it, and the boot sequence involves loading the checksum and the rest of the floppy contents into RAM and computing the checksum, corruption detection would be near instantaneous and the remediation would be as quick as it takes to eject the corrupted floppy, grab another floppy copy from the bin, plug it in, and flip the on switch. So the delay would be (roughly) zero.
Let's say it's not just booting from it, but constantly reading.
Let's say no one implemented the checksum. Edit: I forgot that floppies have a crc check, my bad.
Let's say the machine that can write the new disk is physically far away from the machines that read them. (Or the guy with a stack of new disks -- which are harder to find every year -- his office is some distance from the control machine).
I guess I assumed it was only reading from the floppy, not also writing to it.. that doesn't seem like it would last very long at all without encountering errors (at least based on my memories of reusing floppies until they were completely worn out)
One of my older jobs relied on ancient hardware, and the life of the company was measured by parts in the warehouse and vendors who still cared. From their perspective they'd have a modern system as long as your company delivers.
> hey haven't shown any interest in updating the system.
So expensive to update that there's a calculated end-of-life to the system. They'd love to know about your engineer situation. That'd trigger plans put in place a while ago.
Your company could do a last big batch for the city and send the old guy out with a nice bonus.
Nah getting you fee engineer interested isn't that hard, there's always a fun challenge in mastering something. You just have to give them time to figure it out.
At my workplace, we still ship devices with AMD 8086 processors to the military. And AMD still makes us 8086 processors on special order.
We got a few guys under 40 that work on that project once in a blue moon when a change is required.
Where are you going to find a young engineer who wants to build a dead-end career? I'm sure that if you pay them enough, someone will show up, but most people would rather build skills that will continue to be useful rather than skills that will be useless once an inevitable, sorely needed refresh happens.
I think you overestimate how much working on obsolete tech is a dead end career. Skills are still transferable. Someone who can sling 8086 assembly can figure out your dumb web app.
The replacement will use modern parts and tools, administered by someone who knows modern systems.
There is no use in becoming an expert in an ancient system ridden with proprietary and obsolete parts, just so you can help one city maintain their ancient system. It's even worse than wanting to become a professional fax machine guru.
There is no use for so many things…
For example there is no use in assuming you know what other people want to do with their time, because you just don’t know.
Some people like knowing unique things, or just enjoy learning, or want to be a small hero for a small group of people, or are just curious.
> This system was designed to last 20 to 25 years. SFMTA's director Jeffrey Tumlin said upgrading the system will take another decade and cost hundreds of millions of dollars.
This is the problem, not that they're using a floppy. This isn't web dev where you get to rewrite everything every 6 mos. Systems have to have decades long life cycles BUT THEY EVENTUALLY NEED TO BE REPLACED and that's not happening quickly enough here.
Edit: It was last updated in 1998, so it's due now not a decade from now.
I'd like to see a breakdown of that "hundreds of millions of dollars" and which companies provided the quotes, plus who owns the companies and who are their husbands/wives/uncles/nephews/nieces/etc.
I looked into this last year, a significant part of the cost is in new hardware, both on the vehicles, but also sensors spread through the system. They all need commercial grade RF systems that allow for 2 way signaling communication and integration into the vehicles control systems.
> Grade 0: High Temperature Operating Life (HTOL): +175C for 408 hours. High Temperature Storage Life (HTSL): +175C for 1,000 hours. Temperature Cycling: -65C tp +175C for 500 cycles
And generally speaking transit agencies aspire to buy higher-reliability, longer-lasting equipment than your average consumer car.
I’ve noticed that my car’s touchscreen operates at a very high temperature at all times - almost hot enough to burn skin, which seems like a weird choice for something designed to be touched by skin.
That probably means that whatever is running behind the touchscreen is the thing that’s hot, and the touchscreen is doing a poor job insulating that heat. But is also illustrative of why the automotive standards are the way they are.
It’s probably not overt corruption. It’s probably because in this kind of large system, there is a strong demand for validation and assurance at all levels, which is usually achieved by hiring experts* to write reviews, recommendations, and reports that hold up in court.
* They must be experts, otherwise how could they get away with charging so much, right?
The cost is distribute across all the PowerPoint, Word, and Excel spreadsheets created to give the illusion that people are doing actual work. Go read a couple of the SFMTA PDF's linked to by a commenter. They are totally enlightening justifications like - reduced congestion blah - better monitoring blah. No meat whatsoever. No analysis backing anything up - just pretty pictures.
I rode Muni for 10 years almost every working day. Do you want to know what reduced congestion the most. Buying new muni cars because the old muni cars door mechanism had been repaired so much they did not open or close %50 of the time and people had to use other doors to get on and off a train on a daily basis.
Still, when you have a 20-25 year replacement cycle being 10 years behind is pretty significant. And that's assuming their estimate of needing another decade is correct. Which seems doubtful considering they don't have a contractor, they don't have the money, and the previous system probably took a decade to develop considering their choice of 5.25" floppies (3.5" floppies overtook 5.25" floppies in sales in 1988).
Right, if they think it will take 10 years, 15-20 is probably more realistic, which means you need to start the upgrade process 5-10 years after buying your system: seems very expensive.
What's really happening is going to be:
Emerging new use-cases and rising costs from specialized vendors for replacement parts that match the existing system are driving a desire for the agency to replace the control system. The floppy system bit can 100% be replaced with a solution that isn't a floppy, but that wouldn't help them with the rest of what they want.
>Turns out that in 1998, SFMTA had the latest cutting edge technology when they installed their automatic train control system.
> "We were the first agency in the U.S. to adopt this particular technology but it was from an era that computers didn't have a hard drive so you have to load the software from floppy disks on to the computer,"
In 1998, most personal computers already had hard drives [0]. From Wikipedia "The IBM PC/XT in 1983 included an internal 10 MB HDD, and soon thereafter, internal HDDs proliferated on personal computers."
The 3.5" floppy is from the mid 80's, again from Wiki [1] "In the early 1980s, many manufacturers introduced smaller floppy drives and media in various formats. A consortium of 21 companies eventually settled on a 3½-inch design..."
Why would a hard drive be better? If the floppy fails just grab another off the shelf and try it (surely they have more than one copy). Downtime measured in seconds.
The other good thing about the floppy is it can't hold very much code. So the system has a tight upper bound on how bloated and complex it can get. Simpler systems are more maintainable.
These things seem like great assets for maintaining critical infrastructure.
EDIT: Another great thing is such a system will be stateless. No disks, no filesystems, no databases. Sign me up.
Where is the factual error? This passage is the one we're taking about right?
> Turns out that in 1998, SFMTA had the latest cutting edge technology when they installed their automatic train control system.
> "We were the first agency in the U.S. to adopt this particular technology but it was from an era that computers didn't have a hard drive so you have to load the software from floppy disks on to the computer,"
The fact of when hard drives became commonplace in personal computers (EDIT: or when 3.5" floppies were introduced) has no bearing whatsoever on whether this transit control system was cutting edge in 1998. This statement is not, at least not obviously, factually incorrect. So if you're going to claim that, show your work.
I'm not sure moving it back a whole decade saves them?
My family's first PC in 1987 had a hard disk. The Wikipedia quote they provide lines up with that, and provides a more authoritative point of when it was introduced.
And yeah, the "5 inch floppy" quote paired with a photo of a diskette. God only knows what actual hardware the system uses. But point being … the journalist doesn't seem to have found out.
There's a big difference when choosing technologies for a home computer versus a safety-critical system that will run for decades. Seems reasonable they would've chosen a proven, if slightly outdated, tech over a newer one that had less data to back up its long-term reliability.
And of course we can get into the discussion of cost: floppies are way cheaper than hard drives, especially for the presumably-small amounts of data that are needed for such a control system. A 500MB hard drive was probably overkill.
I am of course speculating, but I don't know why we should assume that the engineers working on the project originally made a silly decision. They almost certainly weighed the available options to deliver the project within the defined constraints of the system.
Hard drives weren't new in 1998. I don't even know where you'd have gotten a computer that lacked one by then; and most would also have had a CD-ROM drive.
Yeah, on further reflection, I recall my parents having a 286 with a hard drive. Still don't get why they couldn't find a 5.25" floppy. Then again both types of floppy drives are still quite obsolete....
Why do I have to do this research instead of the "journalist"?
Ratchet the snark back. The journalist was referring to the train control system not home computers in someone's basement. And, yes, twenty five years ago SelTrac was cutting edge. Moving block systems were basically unheard of back then.
> And, yes, twenty five years ago SelTrac was cutting edge. Moving block systems were basically unheard of back then.
Meanwhile in Germany, we have had moving blocks from the late 80s, based on a technology developed from the mid-60's and production-ready by the 70s [1]. Incredibly, the LZB technology never had an actual accident happen in all the time, only three "bare misses" (one of which was pretty spectacular in that it caused a train to pass over a switch rated for 80 km/h with around 185 km/h without derailing).
SelTrac has been around since the mid 80s, and was originally a German developed product (Standard Elektrik Lorenz) based on LZB technology.
My understanding of LZB is limited, but it appears to be a fixed block system with wayside detection of trains, albeit with shorter blocks than lineside signals would usually have. This is different to SelTrac which is moving block.
Yeah, LZB internally is based on fixed blocks - but on a system with block lengths down to 50 meters (as in the Stammstrecke München [1]), the difference is negligible.
Crazy to see that SelTrac was actually developed in Berlin, failed there, but is still in widespread use across the world.
The only snark here is yours. What are you talking about with "computers in someone's basement?"
He pointed out glaring factual errors in the story, which should not have made it through any kind of editorial review. For example: 25 years ago pretty much every computer had a hard drive. And the disk depicted in the article is obviously a 3.5", not a "five-inch floppy."
Whether the computer in your basement had a hard drive or not has no bearing on whether or not SelTrac was cutting edge at the time (it was). That there were even computers with microprocessors put SelTrac decades ahead of what was in use elsewhere at the time.
You keep parroting this maxim without showing any convincing evidence of an error... what's the point of that? You obviously have some affinity for factual correctness, yet your style of argument and evidentiary rigor appears to boil down to "repeating the maxim many times in many different places makes it more factually true" which is obviously nonsense. Maybe some food for thought?
Given that the agency seems not to be able to cope with change at a reasonable pace, I wonder if 1998 was just when the control system project finished, having been planned and started several years prior.
I had the displeasure to operate a sort of industrial/automotive rack computer running very simple Simulink data logging that booted only from a floppy, and I think it was bought around 2008. For a greenfield project.
Probably cost like hell and was decade or two behind in technology, but I didn't buy it and definitely wouldn't have. Salespeople get people to do stupid stuff.
I'd guess these are used a lot in industrial settings where the code and task is actually very simple. Would run no problem with a $1 microcontroller and a hundred lines of C and with a lot less hassle. Likely vendors have gotten a reputation and keep on selling them with FUD.
Well... I feel slightly better about Boston constantly pushing back being able to use our phones as tickets after seeing the timeline for the transition and how just that they are still doing this.
What is it about public transit in the US that it is so... bad? Inadequate funding seems to be the easy one, but the MBTA (Boston) doesn't even handle the funds it has well. Yeah it needs more funding but there is also just a core issue to how it's run.
It is sad to see the state of public transit in this country, particularly in dense urban areas where we should be discouraging Car use as much as possible.
I am very curious what other countries are doing that we are not.
I am an elected official (12 years and elected six times in Illinois) unfortunately seeing this process work over 12 years is discouraging. For example President Obama allocated 8 billion dollars for high sped passenger rail service improvements. By 2017 almost 2 billion of this money had been spent on railroad improvements / railroad crossings and line inspections.
Fast forward to 2024 no high speed rails have actually been built. For example Chicago Illinois was to get a line from Iowa City to Quad Cities to Chicago (straight line). This project was fully funded at one point but Iowa and Illinois had (still have) a disagreement on bridge repairs connecting the states.
Now Governor Pritzer has authorized a new high speed rail commission to try and get this project going again.
In a nutshell .. Rail project between Iowa City and Chicago was fully funded as of 2016… all that actually happened is some rail road crossing were improved.. lines were inspected and no actual new lines constructed. Instead 1 out of the 2 billion was spend on “engineering” costs and compliance paperwork … which now has expired and need to be redone if the project is to be completed.
The amount of money spent on compliance paperwork and “engineering” is staggering. Many six figure salaries depend on slightly altering existing engineered projects to meet compliance requirements for projects that are never built. From waste treatment upgrades, water treatment, road improvement, traffic studies, on and on. The amount of money spent on services that are not finished or lead to an actual project is absolutely staggering … entire industries depend on this inefficiency and lobby effectively to keep things “obtuse”.
From someone who's seen the other side... every time the engineer has to quabble over email with the permitting departments over matters such as "you sent it to the wrong bureaucrat so I'm ignoring instead of forwarding it" or "the permit admin wrote the wrong 1 of 50 Comcast subsidiaries on the paperwork so now I have to get a skip-manager's signoff" it's billed at $400/hr.
I feel like Boston has had similar, we have contracts with companies to complete a thing by a specific time, they don't but we are stuck paying them even more to complete a project so it ends up being over budget, late, and possibly not done well as is evident by our green line extension that opened last year that was shut down to redo it because the tracks are the wrong width.
It just feels like the money is going to the wrong places like you said.
I would probably argue that we still need more funding even if we fixed how we used the money, but we need to fix how we use it first.
This is what happens when the repercussions for poor performance/failure is more money. Privatization, with some regulations, and reduction of monopolies, seems like a good idea. But, the system will never choose to harm itself.
The current engineering ecosystem is largely private, and it's poorly functioning, costing us way more money than just hiring engineers as public employees rather than paying a contracting firm to do initial and final design on each rail system segment.
Just go look at the RFPs and responses each municipality has four expansion of their current system. It is all outsource to private industry, at great cost compared to doing it inside government.
The lack of accountability happens because the government is not allowed to build up the in-house expertise they'd need to manage projects like this, because of this bizarre American idea that government=bad private=good. So the government agencies end up having to outsource absolutely everything - and not only is that expensive, it's also doomed to failure, because understanding and accountability by its nature can't be outsourced. And that's only compounded by a lack of trust, low-bid rules and the like.
If you look at the entities that have actually managed to get stuff like this done (which is something most developed countries around the world manage), it's not private industries, it's governments - but governments that were enabled and trusted to build-up effective in-house teams for doing large projects.
I think about this all the time every time I see an lcd screen in my local train stations. If people who designed them actually used transit, maybe they'd have the signs you can only see as you walk out of the station say something about the bus lines you'd transfer to, instead of when the train behind you that just departed from is set to arrive. At some stations the screens they installed don't display anything useful at all, not when the next train will come, just the date and time as if everyone doesn't have that in their pocket, and this needs to be displayed every 25 feet on the platform.
A lot of transit could be fixed by just taking a regular routine user, empowering them to become a dictator for a week and point out all the friction points they hit actually using the system. But then that would make the entire bureaucratic system that is the transit agency look like idiots who don't understand their own jobs, so it will unfortunately never be done.
For sure. I think a lot of this is just about how heavy the planning cycles are. A lot of stuff like that is developed through big, waterfall planning efforts and large outsourcing contracts. Most of the choices were made before they had their first user, and once it's all installed any revision would be a) expensive, and b) a black eye for the people in charge.
And in some ways I don't blame people for doing this, because you need really supportive stakeholders to work in an iterative fashion. Otherwise you get a ton of nitpicking that amounts to a lot of "why didn't you guess 100 perfectly everything up front" and "how can we start if you can't give me a full plan and a firm price?" In a blame-hungry environment, waterfall is the safest choice for the people doing the project.
I think a lot of projects have adopted a ritualism around an ideal that only exist in renders vs actual learned pragmatism as well. Despite these glaring shortcomings that I really must not be the only one to realize, the system still creates new stations or other infrastructure beholden to the same shortcomings. Pro transit journalists have their work cut out for them: it's so easy to ding these builds on a usual laundry list of items, but somehow the engineers are like automatons who blindly follow prescription probably from a higher form of government and one again, never seem to test the system beyond simulation I'd presume. How do you fix this? It seems so fouled from the top authority mandating stupid infrastructure to the bottom subcontractors who are hired to implement these plans and use their influence to perpetuate such byzantine planning, where they will be the most qualified contractors to bid on future projects. The inertia of these maligned incentives seems insurmountable. Everything seems so rotten and hard to fix without a total reset.
For sure. One way I'll talk about this is, "The biggest fantasy novels I've read are project plans." Everybody coalesces around a vision quite detached from reality and then just kind of lives in it for years. It's a weird sort of collective arrogance/faith.
There are alternatives, though! In Rother's "Toyota Kata", he describes a Toyota practice where they pick some far-off goal (which I think is called a "target condition"). That can be something that nobody knows how to do. Then they take one step in that direction and reassess. Over time the path might be somewhat wandering as they learn what works and what doesn't. Or they might revise the goal based on new knowledge. But they keep going, step by step.
That's a lot like short-cycle iterative software processes. Which I know work, because I've used them for many projects. But even software, which is infinitely soft, often gets the same sort of fantasy-driven planning.
I think you're right that the ultimate problem is how authority behaves, and at least in the US there's a terrible managerial culture that this is embedded in. My solution has been to find small pockets of sanity, but I've not figured out how to scale that up. But you might find some hope in this Mary Poppendieck talk, where she explains that it hasn't always been this way: https://www.infoq.com/presentations/tyranny-of-plan/
"Which way do I go?" orientation signage in the Boston subway system has been less than wonderful. A lot of "the obvious thing needed here is ..." not happening. Places you could stand to help confused and grateful tourists. So tempting to do a self-adhesive-vinyl signage project. And then the DNC did.
When the Democratic National Convention came to Boston (2004), the subway got a lot of nice "which way do I go? what are these stairs? if not these, then where?" temporary orientation signage. It lingered afterward, with some becoming permanent signage, and others puzzlingly not. Such an event involves massive interdepartmental communication, tight timelines, altered incentives and constraints, and additional resources. Some of the altered communication channels are said to have persisted. I might be interesting to look at events where things are "shaken up", to better understand the steady-state tangle.
The big issue is putting those screens in isn't just a simple task, you need the entire support and switching network to be able to track individual trains which is a relatively new thing in terms of the age of many of the subway swtiching and train tacking (where those even exist) systems in the US.
Honestly a big issue seems to be in other countries, especially ones like Japan people frequently site as the model for public transit, transit is privately run and developed more pragmatically/more profitably. There wouldn't be sense in adding useless screens with useless information, but there sure would be a lot of consideration on how we can make our station more attractive than a competing line's station, paring down overhead to its most base items to further that goal.
Whereas here in the US, its like transit departments are handed budgets generated by tax dollars but no clear direction or plan for how to maximize those dollars. So they have to turn to subcontractors who eat through the pile of money like a vulture eats carrion. This is a system that over time lobbies to further entrench and sustain itself and becomes impossible to rip out and restart. If you go advocating for privatizing the transit system you'd be likened to Judge Doom before people stop and think for a second of what that would actually look like: the shinkansen for example.
Metros would rather build a sprawling parking lot for their bus and train operators to park their cars in, than to ask why the local metro system is such a nonstarter to so many of the metros own employees.
Japan's transit is private and semi-competitive. (there are 10+ train companies in Tokyo), (4+ in Kyoto), I haven't counted Osaka's
It works because the companies own land and facilities around every station. Grocery stores, office buildings, shopping centers (stores at many stations), apartments, etc. This creates a virtuous cycle where the more riders the more people use their other services and visa-versa.
As for semi-competitive, at least in Tokyo, Osaka, and Kyoto there are enough lines that you often have a choice. For example Tokyo (area) to Yokohama there's JR, Keikyu, Tokyu, (3 different companies). They have slightly different routes so if you're closer to one you might take one or the other but they do advertise trying to get you ride their's over the other's. To Hanada there's Keikyu and the Tokyo Monorail (it's own company). To Narita there's the Narita Express (JR) and Keisei (a different company) as well as local lines from both. Same in Kyoto. You can go to Kyoto to Nara via JR or via Keihan. You can go Kyoto to Osaka via JR or Hanshin.
So, if one company offers easy pay methods and another doesn't it quickly gets the reputation and "a crappy old line" (who wants to live there, open an office there, etc...)
I'd love to know what incentives make this stuff work in countries with pure "public" transportation. There the obvious "it's the right thing to do" but this American (me) sees that incentives work and I can't see what incentives keep politicians, which are human just like everyone else, from syphoning funds from public transportation to other things, not able to get a large enough budget to upgrade since they need all the other politicians in, not get influenced by industry trying to sell cars, etc..
- political point-scoring - blame-oriented cultures discourage experimentation and incremental improvement
- political polarization - one-party areas can more easily slide into cronyism, and fighting between parties makes it hard to compromise even on things like fixing infrastructure
- classism - in a lot of places, transit is for the poors
- racism - many don't want transit bringing Those People around
- manager culture, not engineer culture - as we see with Boeing, standard MBA thinking doesn't work well for long-term safety and reliability; the focus on short term metrics, mostly financial ones, leads to underinvestment and decay of infrastructure
I’m a liberal democrat, but it’s clear that one party politics is not tenable. There has to be an option to the status quo that you can vote for to incentivize competency in government. But with the state of national politics it would be almost impossible for a republican to win in most of urban America. So that leaves just primary challengers from within the Democratic Party which is very difficult. So urban political machines have become absolutely rotten with complacency and now the richest zip codes on the planet have trains running on floppy disks.
I totally agree we need more actual competition, and also agree that national polarization plus a first-past-the-post system makes that unlikely in the near term.
I would also love to see widespread use of RCV, but as a San Franciscan I don't think it has done tons to fix the problem of complacent political machines here, so we're going to have to do more.
I struggle to understand how unions/government are often seen as the problem. Some of the best public transit I've experienced have been in places with strong unions and more government-owned/managed public transit e.g. much of Europe. Some of the worst in places where unions are weak and "public" transit is largely out-sourced to private entities operating on behalf of public transit agencies e.g. the US, UK, Australia, New Zealand, etc.
I live in New Zealand public transit in NZ used to be incredibly bad during the heights of private operation in the 1990's but has improved heaps over the last 10-20 years as local/central govt have progressively taken back control over more aspects of public transit. However public agencies are still not allowed to directly operate bus or train services which still sees private companies failing to provide good service with no consquences and public agencies unable to take over to provide the service improvements that are needed. It is interesting to compare New Zealand in the 1950's had one of the highest levels of public transit usage per capita under public ownership of buses and railways. This dropped down to embrassingly low levels after out-sourcing to private entities to a low in the 1990's.
It has only been under government leadership that we have seen a revival to the current levels (e.g. doubling to 100+ million trips per annum in our biggest city from well below 50 million in the mid-1990's) due to local government coordinating public transit systems (and contracting private entities to operate to the timetables set by public authorities) that meets the needs of the users and not just private entities making a profit. Unions have been vocal too for improving systems not only for transit employees but also recognising that what's best for transit users also benefit transit employees.
YMMV and possibly depends on the political environment in each country.
public/privately operated transit is a completely different topic than unionized/non-unionized staff except in the west where public sector unions have the government by the balls
Consider the cost of Singapore building with imported migrant labour vs NYC building with ~~mafia~~ union labour
> What is it about public transit in the US that it is so... bad?
Massive money and entrenched behind preserving a car-based life.
Car companies, auto workers unions, railroads can siphon off funds from promised improvements to cover deferred maintenance and give themselves bonuses for being so clever… Hell, AAA which one might expect to spend its income on serving its members instead diverts money to proactively lobbying against improvements in public transit in an ongoing display of cynical self-preservation
Nonsense. There is working transit in the US and a huge appetite for more. But it takes decades to do this stuff. "China did massive infrastructure project in two years" -- China is also a communist dictatorship (for some value of those words), and we can't do that here. We could streamline things a bit more, of course, but it's never going to be cheap and easy in the US.
If you want to help, vote. Or even run for something on a pro-transit, YIMBY platform.
> We spend more than any nation on public services and goods.
In part, because of the ideology the poster outlined. Penny wise, pound foolish.
As an example, our healthcare is 2-3x as expensive as the rest of the OECD. Our public spending is more than anyone else's total spending, and then we pile on top of that. (https://data.oecd.org/healthres/health-spending.htm) We saddle the poor with high-deductible plans they can't use, then eat the costs of their eventual last resort emergency treatment.
When BART was built in the 60s/70s it was innovative. The construction techniques for the Transbay Tube were unique at the time. The system was billed as a transit system built from the ground up for the space age. It was equipped with, at the time, a state of the art Automation Train Control system.
Upgrading those systems is expensive and difficult, so it's not surprising old prices of equipment hung around until the end of their useful life. Especially since BART is going through a massive budget crisis at the moment (and this isn't the only one in its history).
I've shared this experience/opinion with folks nearly verbatim :) Especially after having seen what I took to be a very well-integrated transit system in Chicago, the fragmented and confusing Bay Area experience was quite different from what I expected.
SF Voters rejected Proposition A in 2022 [1], which would have included funding to upgrade Muni's control systems (among many other projects). We'll eventually have to find the money somewhere else when the system fails.
The entire system has reached the end of it's design life and the floppy is just the shocking bite to get people to click through. Really this is the regular replacement of an old system with a newer control system that will make the trains run better and more reliably.
Why will a new system make the trains run better? It could also make it run worse, if they don’t cover all the current cases.
I’ve ridden Muni in SF for years, drivers control most of the decision making.
There’s a handful of single track tunnels/sensors/necessary software-based coordination, but as another commenter pointed out, the doors cause more issues than signal problems.
I’ve ridden Muni in SF for years, drivers control most of the decision making.
All of the tunnels are under automatic operation, and the majority of the switches (street or tunnel) are computer controlled.
the doors cause more issues than signal problems.
While doors are a real common failure/abuse point, I'd want to see some actual numbers on that. Train control failure may not always be self-evident to a rider, especially if the remedy is to run things manually (such that things aren't ever stopped completely).
Historically the cab signaling system would glitch in wet or damp weather. Transponders for the switches had a near 100% failure rate. Trains chewed up the inductive loops, forcing emergency braking. That was common enough that they had to dial back the emergency deceleration. Twenty years on they've mostly worked out the kinks but it hasn't always been that way.
Can someone just ask Foone or lcamtuf to help them get it running off an internet-connected toothbrush or something? It seems like there are so many people doing complex reverse-engineering of ancient stuff _just for kicks_ that this floppy-disk issue just shouldn't require a massive project.
yes, I know safety-critical systems are different. I also expect that the floppy-disk issue is just the easiest problem to explain of a long chain of terrible legacy lock-ins. However, if they're literally holding their breath every morning when it's time to IPL the system off a floppy...that part sounds solvable.
The hypercompetence of tech enthusiasts and the mind-numbing idiocy & dishonesty of politial governance don't mix, so cities have to spend tens of millions with parasitic contractors for what a few clever hackers could do as a side-project.
> Turns out that in 1998, SFMTA had the latest cutting edge technology when they installed their automatic train control system.
> "We were the first agency in the U.S. to adopt this particular technology but it was from an era that computers didn't have a hard drive so you have to load the software from floppy disks on to the computer," said Mariana Maguire, SFMTA Train Control Project.
> SFMTA's train control system relies every morning on 5 inch floppy disks.
This doesn’t make any sense. 5.25-inch floppy disks and no hard disks was not “cutting edge technology” in 1998. It arguably wasn’t even “cutting edge technology” in 1988
Not cutting edge personal computer technology, obviously, but was it not cutting edge train control technology? TFA isn't making any claims about PC technology. If I had to come up with a computer system that needs to last O(1 century) I'd avoid disks, I think.
Frustratingly, though the article goes into how replacing/upgrading the whole control system will be an big, expensive project, they don't attempt to say why they can't just update the "read from floppy" part of the system to read the same info from a modern component.
Floppies aren't the problem, the main problem is lack of support. The MTA moved off of OS/2 back in 2008. Yeah, halfway into its intended lifespan Alcatel/Thales basically stopped supporting the whole mess. If I had to guess they're using an equally unsupported version of Windows now.
Yeah, the article reads as if the real problem is a lack of a tested system of backups. Or I don't know how else to parse this line:
- "It's a question of risk. The system is currently working just fine but we know that with each increasing year risk of data degradation on the floppy disks increases and that at some point there will be a catastrophic failure."
"Data degradation on the floppies" should not, by itself, cause any sort of "catastrophic failure" in a sound system. If one software disk fails, you should have five identical copies in a drawer, five more in a different room, ten more off-site, plus a disk image on cloud storage that you can write fresh floppy disks from.
I mean, it's probably a good idea to change the storage medium too—but that's not the root problem.
I believe that the article is about this RFP for “Contract No. SFMTA-2022-40 FTA” to upgrade the Communications-Based Train Control System (CBTC) or Advanced Train Control System (ATCS). The resolution to create the RFP for the supplier was approved 2023-01-17 (https://www.sfmta.com/reports/1-17-23-mtab-item-14-communica...), and a $36M resolution to create an RFP for a consultant was approved 2023-11-07 (https://www.sfmta.com/reports/11-7-23-mtab-item-11-train-con...). I’m not sure where the actual RFPs are pubished though.
SF is a joke. 20 years of being one of the richest cities in the world (probably in the history of human beings) and it comes out worse than it was before.
Upgrade? More like launch the process of planning for drafting a bid solicitation announcement. If I were 3.5" floppies, I wouldn't worry about retirement any time soon.
"The San Francisco Municipal Transportation Agency (SFMTA), which runs the city's Muni Metro light rail, claims to be the first US agency to adopt floppy disks. But today, the SFMTA is eager to abandon its reliance on 5¼-inch floppy disks—just give it about six more years and a few hundred more million dollars."
> "Wow. I mean I thought we were moving on to AI. So why are we doing floppy disk," said Katie Guillen, SFMTA passenger.
This naivety is not Katie's fault. We who work in tech are to blame for constantly pushing our half-baked experimental garbage as if it was "engineering" on par with civil or aeronautical systems. We can't blame people for occasionally believing the lies.
The tech-washed version of this quote might go something like "wow I thought everything was moving to the multi-cloud serverless kooberneetus now, why is it still running on a computer?"
> It is easy to run a secure computer system. You merely have to disconnect all dial-up connections and permit only direct-wired terminals, put the machine and its terminals in a shielded room, and post a guard at the door[1].
This is the kind of thing I want running the trains. Give it ECC RAM too, please.
> "It's like if you lose your memory overnight, and every morning, somebody has to tell you hey 'this is who you are and what your purpose is what you have to do today,'" said Maguire.
Yeah, that's called a cold boot. Moving to not-floppies doesn't mean you can avoid this. Clearly it's off of floppies instead of ROM so you can more easily update the software, but I am wondering how often that ended up happening. Maybe EEPROMs would have been better.
> Luz Pena: "How dire is it to change the system to upgrade it from a floppy disk to a wireless system?"
I agree that floppies aren't the peak of reliability, but "a wireless system" also sounds like a disaster. I don't want critical urban infrastructure running on extremely hackable OTA updates. For the love of god, SF, you can avoid pretty much all potential cybersecurity problems by just not putting your trains online.
I feel like neither the interviewer nor the interviewee really had the technical expertise to speak to this. This entire piece is just, "oooooo, floppies are old. Old bad! Why not new yet? New good."
I don't think that is actually a concern. The article is written in a way to imply that it is, but I doubt the people who are running the system actually think that is the risk.
I am sure they have copies of the floppy disk, and likely have images on other machines that can be used to create a new floppy even if all the copies are lost.
I feel like the SFMTA is mainly using the existence of the floppy disks as a marketing point for their desire to get funding to update the system. It is something that the average person will be able to look at and know it is out of date in a visceral way.
The reporter seems to read this as if we are one bad floppy away from failure, but that is not the actual case.
You are correct that this would be much simpler if the only goal was to replace the floppy disk part of the system.
The floppy disk angle is there to make a good headline. The article makes it clear this is a much bigger project than just replacing the floppy disks.
"The detail[ed] project schedule will be finalized once we have a contractor onboard. This is effectively a multi-phase decade long project that starts with pieces of market street subway and pieces in the surface. Ultimately our goal is to have a single train control system for the entire rail system," said Tumlin.
SF has to make everything a big bang project because every project has equal large fixed cost. So whether you do a small thing or a big thing you do a big fixed cost thing first.
So SF always does the big thing because otherwise the overhead dominates.
Playing video games on an old Apple II via a floppy emulator has a different set of requirements than a safety-critical application.
Lawyers will probably spend a year wrangling liability concerns.
I could probably replace a floppy drive on any system within a couple of weeks. I would not accept legal or financial liability for any such solution without an extremely thorough and slow review of all aspects of the project. edit: and an astronomically high paycheck.
> However, budget challenges put the project's timeline into question. The SFMTA's train upgrade project isn't just a migration off of floppy disks but also a "complete overhaul of the current train control system and all its components, including the onboard computers, central and local servers, and communications infrastructure," Roccaforte said.
> Much more critical than the dated use of floppy disks is the system's loop cable, which transmits data between the central servers and the trains and, according to Roccaforte, "has less bandwidth than an old AOL dial-up modem."
Also, they're already late as hell, what's 5 more years? 5 1/4 isn't going to become more obsolete than it is at this point.
You don't need more "bandwidth than an old AOL dial-up modem" for train control. What Roccaforte left out is that the MTA spent over a decade trying to keep the trains from slicing up the inductive loop. No idea how the new trains are working out, but I'm pretty sure they never found a permanent fix for the old ones.
There are other concerns that need to be addressed where a hack wouldn't help:
>The transportation body says the train control system was built to last for just 20 to 25 years, meaning it surpassed its expected lifetime in 2023. In 2020, the Muni Reliability Working Group, said to be composed of local and national transit experts, recommended replacing the transit control system within five to seven years. [...] "We have to maintain programmers who are experts in the programming languages of the '90s in order to keep running our current system, so we have a technical debt that stretches back many decades," Tumlin told San Francisco's KQED in February 2023.
I think the government is involved which means the task must primarily be an effort to funnel large amounts of money around to just the right people while facilitating all the best changes to whatever they can find within N degrees of separation from the project itself, (where N approaches infinity.)
Speaking naively (I have no insight into the situation), I would guess that this is probably the result of the officials in charge not having technical understanding and contracting it all out. Contractors aren’t going to be choosing the quick or cost-effective option, because that’d undercut their profits.
Also, it is difficult to sell a solution of "well - we aren't really modernizing the system, we are just putting a fix in place that pretends like it is a floppy drive."
I would guess higher ups don't want to spend money on a band-aid without really fixing the problem.
>""The system is currently working just fine, but we know that with each increasing year, risk of data degradation on the floppy disks increases and that at some point there will be a catastrophic failure," Tumlin told ABC7."
They haven't even settled on a contractor yet? Maybe the problem here is that they're trying to write one check to fix all their problems at once instead of taking an incremental approach.
This takes deep expertise and a system oriented, long term view with a very precise eye towards the details of what's technically possible at every individual step.
This kind of knowledge and experience doesn't come cheap, and we all know how much US city governments pay. I was at one point, very briefly, motivated to apply to a city job before I learned the pay is approximately 1/3 to 1/4 of what the private sector pays. The USDS routinely posts on "who wants to get hired" and the comments on that and other thread also mention a 66% to 75% salary reduction from baseline.
This is even before we get to legal liability and political risk shifting that can happen when there is a fully responsible contracted company involved.
I get it though, it's a nightmare getting a project through approvals so by the time you have an issue big enough to push an overhaul through it's a major project and you want to pile as many things onto it as possible because it might be a decade before you get another decent funding injection to do what should be regular maintenance.
The longer a thing has not failed, the less likely it is to fail at any given moment. On the other hand, the older a thing is, the more likely it is to fail at any given moment.
They do use 3.5 inch disks; this is a misstatement: “SFMTA's train control system relies every morning on 5 inch floppy disks.” The SFMTA Board presentation justifying the >$600M upgrade shows a photo of the real 3.5 inch floppy disks in use that were written in 2021. https://www.sfmta.com/sites/default/files/reports-and-docume...
Trains are expected to last for decades, so this really isn't surprising. Developing a new system with the latest hardware to replace the floppy disk system would probably cost millions and might introduce bugs and security issues.
Just a few short hours ago I was hearing some Muni workers at West Portal station talk about this story. One said to the other, "young people don't know what a floppy disk is anymore".
Raspberry pi controller $30, couple of people to make it work - let's say $500k.
Sure... there are loads of things that probably need attention, but maybe you don't need to fix them all at once?
Remember when, less than 3 years ago, SF projected a $108M surplus for the next two years?
Woulda been a nice time to clean up some of this technical debt!
Or how about the SF Emergency Sirens, taken offline in late 2019 for a "2 year" upgrade plan that officeholders implied was already in place?
In August 2023, with no progress whatsover, with the Maui fire disaster fresh on their minds, Mayor Breed & Supervisors President Peskin touted they'd finally funded a plan to return them to service soon: https://www.sf.gov/news/mayor-breed-and-board-president-pesk...
Of course, this was just more blatant self-exonerating bullshit from our local political machines immune from any real accountability for incompetence in basic public functions.
A mere 6 months later in February 2024, nothing's been started, Peskin admitted "we don't even have a plan", the department is still waiting until "funding is identified", and the cost estimate has ballooned to $20.5m: https://abc7news.com/san-francisco-sirens-emergency-911-aler...
That works out to $170K+ for each of 119 units – units that each could probably just be a weatherized consumer-grade handheld device with multiple mobile/packet/sat radios, & a simple authenticated-playback app, mounted on existing poles that presumably already have power and even loudspeakers.
Sirens don't even need to be that complicated. A motor, a noise generator wheel (basically just a big spinning squirrelcage fan), and a horn. You could toggle them on/off with a simple relay/contactor.
These "sirens" actually need to be able to play spoken announcements - something they did easily since about 2005, before the fumbled 2019-2024 shutdown/upgrade.
More history, through January 2022, via JWZ's blog:
Yep! And as of the shutdown over 4 years ago, even if their control plane was insecure, they presumably had 119 working loudspeakers of the proper loudness/weatherproofing/etc.
> It's a question of risk. The system is currently working just fine but we know that with each increasing year risk of data degradation on the floppy disks increases and that at some point there will be a catastrophic failure.
The key words in the sentence are: "it's working just fine".
Data degradation of floppy discs is easy: just copy them to fresh ones, and verify that you have a good copy. The images should be safely backed up so they can be regenerated. (Plus there are emulators; a topic covered elsewhere under this submission.)
I mean, are they really using the same 30 year old floppy discs over and over again until they degrade?
They emailed me back, they said that the floppy thing makes a good headline but is really just the tip of the iceberg. It's really the whole system that's like this at every layer, it needs replacing they say.