Getting paid to have fun

How to get paid to have fun

After the wireless alarm debacle, I went to work for a small company for about a year before going off to Uni. The company built exhibition displays, training aids and simulators and it was a pretty cool job. I was employed as the "boffin" who had to design all the electronic gubbins that made all the things work, and every job was a one-off.From the day I started I was thrown into a variety of weird and wonderful projects. This page is about a few of them, particularly the failures!

Security schmecurity

The Vanguard-class submarine

Pretty much the first job was a full-scale working mockup of the machinery control room for the Vanguard-class submarine. This was Margaret Thatcher's Cold War deterrent, built to carry the American Trident missiles into battle, should it ever come to that. The subs were just being built, but the first wouldn't be commissioned until a few years later. The Royal Navy needed a way to train crews, and part of that solution was the simulated control room. In theory this project was pretty much top secret, but since I was never called upon to sign the Official Secrets Act or anything else that would force me to keep my gob shut, then I can tell you all about it here ;-) In reality there's not too much to tell, except that we couldn't work from the real sub's schematics. Instead we had written descriptions of how all the systems worked, what gauge, light or dial had to show in response to what, and drawings of the layout of the real thing. My job was to take those descriptions and sort of guess how it all worked, implement it in whatever fashion was needed, then oversee the build and test. Much of the system had to do with how electrical power was distributed around the sub, sourced from the nuclear powerplant, diesel generator or batteries as needed. Another big part were the control and safety systems for the nuclear bit. The solution was implemented totally in hardware using standard CMOS. The multi-mega-amp power busses were simulated using 5v logic rails, and simple logic levels used to represent the presense or absense of power. There were many analogue quantities involved too, all dealt with using dc amplifiers of various kinds. All in all, quite complex. The control room had maybe a total of about 70 meters for all sorts of quantities, and there were barely more than one or two the same. The company had built a similar control room for the previous class of subs, so they had experience of this. All the guys in the drawing office were dreading the work involved in creating all the artwork and screen masks for the meter dials - it was both repetitive and dull yet called for very high accuracy and attention to detail. In their terms it was many weeks of work. When I learned of this, I suggested that the dial artwork could be very quickly turned out on my office perk - a brand new Mac SE/30. The boss thought it was worth a shot so I produced one using Claris CAD at 200% actual size. This was then printed on a LaserWriter, reduced to actual size photographically then silk screen printed. The result was as good as hand created artwork, so once that was proven, I turned out the remaining scales in half a day. I could never tell if the DO were grateful or felt threatened by the little squat Mac in my office...

The drawings and specs of the sub were supposed to be kept in a locked safe in the boss's office, which they were, but we had numerous photocopies lying about and the simulator itself was pretty big, so just sat there in the middle of the production floor for the several months it took to build and test. The building had no alarm and just a standard yale lock on the side door . It was a classic case of "hiding in plain sight". The contrast to the final installation deep inside the Greenwich Naval College in a secured room with several levels of security access was striking.

The bouncing doomsday clock

“Doomsday” clock concept

In parallel with the sub project was a much simpler job to create a huge clock for the reception area of a nuclear power station. The clock was several feet across and had a single sweep hand that revolved once a minute. Each day a 7-segment display counted the number of days since the last industrial accident (the things some people want!). The sweep hand was mechanical but also traced out a circle of LEDs, one per second, which were reset at the 12 o'clock position. This project slightly predated my arrival at the company, so the boss already had a design sketched out. It was a crock, since it relied on using a potentiometer to carefully adjust the voltage on a dc motor that drove the sweep hand to keep it in sync with the digital counters that counted out the seconds, minutes, hours and days that did the rest. There was no way that would work for more than a few hours at best. It later turned out that many of the company's existing works used this kind of approach, and they saw nothing wrong with the constant need to go out to sites to reset or fix displays that had become desynchronised in various ways. My philosophy was that I didn't want to end up doing a lot of that, so it was better to design stuff to work reliably for long periods instead. They found this quite a new concept! Anyway I redesigned the clock using a stepper motor so it could simply be clocked by the same circuitry that did the counting, and also had some power-on reset stuff that drove the hand to a known position before enabling the counting. So we built the clock and of course in terms of its basic functionality it worked as expected. What I hadn't figured however was that the perspex sweep hand was nearly two feet long and consequently quite heavy. The gearing of the stepper motor had the necessary torque but the inertia of the pointer between steps was another matter - the thing bounced and vibrated its way around the clock face in a horrendous fashion - in fact the rotational resonance of the hand was almost exactly at the stepping frequency. This unanticipated effect called for a quick solution as the thing was going out the door next day. So we simply bought some very thick grease from the garage across the road and filled the gearbox with it. This damped out the vibration pretty well so it was deemed "good enough". Unfortunately the clock was returned to us about six months later with a very nasty greasy stain all down its face leaking from the shaft of the gearbox. Bugger. The whole face had to be scrapped and remade, but it gave me a chance to properly fix the bouncing hand. The solution was easy - clock the motor from a different tap on the digital divider and compensate using a different mechanical ratio in the gearbox. Problem solved.

That sinking feeling

Our company did a fair bit of work for the Royal Navy, in particular the PR/Recruitment department. One of the displays I worked on involved constructing a huge fibreglass "pond" made to look something like the dockyards at Portsmouth. A pair of remote-controlled MTBs could be driven across this pond in a race between docking sites on each side. The idea was to have this as the centrepiece of a Navy recruitment display where members of the public could come and race the boats and while engrossed in this could be enticed to sign their life away as a naval recruit. The whole concept was a much larger version of something similar we'd done for another company involving a single model boat raced against the clock. That first effort had worked but had shown up a number of problems that we sought to fix on the naval version. First, we'd used standard 27MHz radio control gear which turned out to work very poorly in many exhibition halls - the interference from fluorescent lights, security systems and walkie-talkies made it very unreliable. Secondly the boats went through batteries very quickly, and thirdly the detection of the boats arriving and departing the "docks" was done using microswitches which were hard to make work reliably, often requiring several "dockings" to get them to register. So the Mark 2 was a rare opportunity to have another go at the implementation. The pond itself was much larger - so large in fact that it had to be made in two pieces. This involved needing a watertight join between the halves which was quite a challenge for the mech boys that had to build it. The solution was to build the steel flanges on a slight angle with a sandwich of neoprene between. The flanges were bolted tightly together and then the pond filled with water. At first the join would leak profusely, but as the weight of water built up, it forced the flanges tighter and tighter together eventually sealing it. Clever. The next problem was making the remote control reliable. Instead of using radio, I used infra-red LEDs hooked up to standard radio control encoders so the signal was directly beamed as digital pulses. IR transmission towers were placed at the four corners of the pond, each using multiple LEDs to give a 90° field of coverage. All four towers were driven in synch. No matter where the boat was and what its heading, it could see at least one transmitter and so always had a solid signal. This approach also required far less receiver power helping to extend the battery life of the boats. We also used much lower power drive motors in the boats - this also was supposed to help lend realism by making sure the boats weren't too fast. Finally the docking reliability was handled using optical sensors instead of mechanical ones.

The electronics had all been tested and debugged, the pond built and tested, the software component that timed and managed the races all programmed, tested and integrated with the dock sensors. It was all set. It was Christmas, and the first use of the display in public would be the Earls Court Boat Show, starting January 2nd. The boats themselves were being built by a modelmaker outside of the company but so far we hadn't seen them. We were assured they'd be on time so it was simply a case of fitting them out with the electronics and batteries. So I had a few days off over Christmas. The responsibility for fitting out the boats was left to the technician who worked for me. Next time I'd see the whole shebang was at the exhibition hall on the morning of the install day, New Years day. Confidence was high because we'd actually managed to get lots of it really well tested and even productionised to some extent - each receiver had a proper PCB and everything. The pond was bolted together, seemed to be in place and so the water fill was begun. Well, guess where the technician had sited the electronics for the control side of the system? Yep, right under the leaking flange. So that got a soaking. We quickly realised the problem and crawled under while it was filling and moved them. The dripping seals and creaking timber frame meant that I really hoped there wouldn't be too much call to have to get under there too much! The boats had arrived and they were very nice - very well made detailed models of a particular kind of boat (I forget what exactly). The electronics had been fitted and given a dry test - all OK. What nobody had thought of up until then was to try actually floating them. That's when I got that horrible sinking feeling. I hadn't specified an exact battery to use - just its electrical characteristics, so the tech had gone out and bought the biggest he could get - ostensibly to maximise the running time per charge. Trouble was, they were very big and heavy, and fitted to the boats just about filled up the hulls. In the water, they did float - just. The waterline was about 2mm below the gunwales! They didn't look right and the Navy bods were pretty unhappy about it, but they did work and time was short, so we had to accept it for now. Next problem was that the boats were now too low to trigger the optical beam in the docks, so we had to add a masking tape "flag" to the mast to make sure it did. Another ugly hack. Anyway, it wasn't ideal but it worked alright, and after a few test races, all was ready for the great unwashed masses.

I knew from experience that if the "public" could find a failure mode for anything you built, they would. So we were keen to hang around discreetly and make sure nothing went wrong. The control helms weren't a concern because they were real boats' helms, built very strongly. The boats themselves were another matter however. Of course, boys being boys (and it definitely was nearly all boys) they soon got bored with the docking races as the game was designed, and instead took great pleasure in trying to smash the boats into one another. The models didn't stand up to a great deal of this, with all that carefully modelled detail soon breaking off. Much, much worse however was the fact that it was soon discovered that a carefully aimed broadside would tip the opponents boat over enough to ship water and from then they had only a few seconds before they would sink altogether. The docking game was forgotten and a new game of "water jousting" became a star attraction of the show! At least until all the boats were at the bottom and none were working due to the electronics being well and truly waterlogged. It started to turn into a nightmare. The Navy said they couldn't shut down the display, so me and the tech ended up spending the rest of the day rescuing each boat as it sank and frantically drying it out with a hair dryer. The public weren't impressed with the constant intervention of the boys in blue, and with the fragility and sinkability of the boats. Nerves and tempers frayed. The navy personnel in charge were officer types used to barking orders and having someone "make it so", and they didn't take kindly to us telling them, well, it just can't be so! Sorry, and all that, but ya canna change the laws o' physics, captain! I copped the most flak and from both sides - my boss was applying heavy pressure, and so were the navy guys. The show thankfully closed at 6pm or so, and we could at least stop the firefighting and work on a real solution. It was obvious what we needed to do - a) get the right batteries and b) seal the goddam boats! Our factory was located about 150 miles from London, so we took the boats and the boss drove us back. We pulled an all-nighter to refit the boats with much smaller and lighter batteries, environmentally seal the circuit boards in case we couldn't avoid the occasional dunking, and also try and seal the boats altogether. At 5am we were done and piled back in the car back to London. By the time the doors of the show reopened all was fine, and everything worked as it should have done. We still had to hang around most of the day just in case anyone needed to do any shouting at anyone, so needless to say we were utterly shattered by that afternoon. I hadn't slept for over 36 hours straight, and had spent most of that time very stressed out. I must have slept for three days when that episode was over.

The Navy only ended up using the display a few more times after that. It turned out that the public almost always ended up water jousting instead of playing the game properly, and soon got fed up with naval types intervening when this happened. As a recruitment aid it was a sorry failure. Still quite a fun thing to build and design however, though I wouldn't want to repeat that first public day ever again!

Hopeful electronics

I mentioned that the public would always find failure modes and also that many of the company's existing works used dubious design practices. An infamous example of this was a relatively simple display that explained how the basics of a nuclear power station worked. It was deployed in the visitors' centre of the power station at Hinckley Point, I seem to recall. Anyway, all this display showed was how control rods worked - you'd press a button and the rods would rise, the pile would get hotter and a meter showed more output. Press another button, the rods descended and the power reduced. Most of the display was of a classic type that worked on the principle of the moiré interefence effect. A blind consisting of alternate black and transparent stripes was constantly moved across the surface of the artwork which was reverse printed on perspex. Slanted stripes on certain areas representing pipes, etc. cause the illusion of movement in a certain direction. This represented the flow of steam and electricity, etc. The only other part of this was a second blind that carried the "control rods". This was moved up and down by a motor connected to some circuitry wired to 'up' and 'down' buttons. This circuitry was a prime example of "hopeful electronics" - i.e. it worked in the factory a few times, and that was deemed enough. To detect the top and bottom limits of travel, the circuit used a pair of voltage comparators. The blind's drive motor was connected via pulleys and a rubber belt to a pot, and this fed the comparators. So when the comparators saw a particular voltage from the pot, it assumed that the limit was reached and it would control the power to the drive motor accordingly. The pot moved over a range of maybe 90 degrees for the blind travel required, which didn't give it a great deal of precision. But, once it was set up "just so" it worked within reasonable limits - after all, the blind wasn't required to stop in exactly the same place every time; within a few millimetres was OK. Trouble is, mechanical things tend to have inertia - as we found with our heavy clock hand.So with the stopping and starting of the motor, the drive belt to the pot would slip a tiny bit every time, gradually putting the whole thing out of "calibration". At first this merely resulted in the control rods ending up parked in the wrong positions, but eventually the pot would end up jammed against one end stop or the other, at which point feedback control, such as it was, was lost.The blind motor would never be given the signal to stop, so it didn't - it would carry on driving the blind one way or the other until it ended up wrapped around the roller at the top. At which point we'd usually get a call-out. So far this was all our own fault (poor design), but one of these call outs gave me an opportunity to witness first hand the sort of abuse that these displays came in for, and how that revealed another obvious failure mode. This callout was during public opening hours of the centre, and a school was visiting. So we fixed the display pronto but thought we'd stick around to see what happened when schoolchildren started prodding. Well it wasn't long before the ADHD-raddled infants were poking and prodding and jabbing at buttons randomly. Hit the 'down' button during an 'up' sequence and the direction was reversed instantly - ok, as intended, but that rapidly threw the pot out of calibration. Within 5 minutes of this treatment the blind was wrapped around the roller again. We decided to address the problem more radically by replacing the rubber band with a toothed belt, which in theory couldn't be thrown out of adjustment. This required some reengineering and a return visit. It seemed to work, until the next coachload of demons attacked it. The next unanticipated design fault soon reared its head - that of the buttons being held down and held down together. It seemed obvious to me that kids would do this, but the display's designer had missed it. The circuit responded to this input by ignoring the input from the pot altogether, and continuing to drive. The outcome was the same as before - the blind wrapped around the roller - so it's likely that prevous incidents had been a mixture of user input and calibration drift. Our "fix" for the calibration problem now made things dramatically worse however - the pot would hit its end stop but now the belt was unable to simply slip. The motor was actually quite powerful - the blind was quite heavy, large and had a substantial amount of friction. So something had to give - yep, the mounting of the pot! This was a piece of folded aluminium plate which gave way, severely damaging the display. It was embarassing to watch it self-destruct in public, but there was nothing for it but to take it away and build a circuit that worked properly and anticipated false user input. Easily done, and after that we never got called out to that display ever again - ah, bliss! The pot feedback mechanics were replaced by a pair of slotted opto-couplers - theblind edge ran in the slot and the limits were set with black painted areas as necessary. It was infinitely more reliable with no moving parts apart from the blind itself and required no calibration as this was effectively imprinted in the blind design. To their credit the boss and technician immediately understood the value of this approach and from then on all displays that had similar moving blinds used the same system, and the number of callouts dropped to virtually nothing.

Invisible electronics

One commission we undertook seemed a great idea at first but soon turned into a nightmare. The customer was a major diesel engine manufacturer, and the requirements were to dramatically illustrate a new fuel injection system. They didn't want a "classic" moiré type display as these seemed very old hat by then. Another type of common display we used for illustrating flow was the "running light" type of effect, where a repeated sequence of three lights or LEDs was switched in turn to create the illusion of movement. The boss had recently seen a new type of sign which consisted of transparent perspex with a design etched onto the surface and edge-lit. The etched parts would scatter the light and seem to glow with no obvious source of illumination. It was decided to combine this with running LEDs to create a dynamic transparent display. Oh, and naturally, it was urgent - we had about two weeks to do it.

Some LEDs yesterday

The display consisted of six sheets of 15mm thick perspex in a steel frame. Each sheet carried a different part of the design, with the front and back sheets blank for protection. One sheet carried the "components", another the fuel lines connecting them, another the text annotation and one the LEDs. The effect was OK - when edge lit it was quite an effective three-dimensional display. The problem was how to connect the LEDs with no visible wires - the boss hadn't considered that, and I got a look as if to say "ummm, whoops. Oh well, you'll figure it out". The sheet that carried the LEDs had some thin (1mm) black lines screenprinted to delineate the edges of the lines, so I suggested we could hide the wiring behind these as long as it was ultra-thin. So I ordered the thinnest enamelled copper wire I could find which was something like 40 gauge - about 0.2mm thick. Being soft copper wire its breaking strain was pretty low - you could pull it apart with a very gentle pull between two fingers. So it would need a delicate touch to use it, that much was clear. One lunchtime I got a call from the chap 'downstairs' who was trying to do the wiring. He was almost in tears. He just couldn't do it. I went and had a look and he'd wired about five LEDs and that had taken him three hours - there were about three hundred altogether. The problem was not just snagging the wires - which was easily done - even soldering them often broke them. No-one wanted to do the job and since it had been my idea I ended up being asked to do it myself, which I guess was only fair! It was incredibly fiddly, there's no doubt about it. It turned into another very late night effort, but eventually it did get done and did work. Despite my best efforts though, it just didn't look that neat. The wires weren't invisible, and every blemish and misplaced wire was very obvious given that the display was completely transparent. It was also virtually impossible to rectify anything without breaking more wires, so once it was working, the boss decreed that it must not be touched.

Delivery day, and we carefully manouevred the finished display onto the company van. It was the worst combination - big, bulky, incredibly heavy yet desperately delicate. The trip up to the exhibition hall was incident free but the delicacy and bulk of the display meant that it took much longer to reach its destination than we'd expected. That destination was a corporate hospitality suite at the NEC where the launch of the new fuel injecton system was officially taking place as part ofthe Motor Show - that day! So it arrived late. The person in charge of the event had also commissioned the display, and had not yet seen it. She was already exceedingly annoyed that we were late as proceedings had already started, and when she saw the finished display, her previous stressed state paled beside its new stratospheric proportions. She hated it. The whole thing - not just the concept, but the transparency, the edge lighting, the colours we'd used, the running LEDs, the less than perfect wiring. Clearly what she'd envisioned was something quite different, even though she'd gone through it with the boss in great detail. In a fit of rage she threw a sheet over it and told us to park it in a cupboard, then after some more shouting, we were permitted to leave. We returned later to move it to the public exhibition stand for the show. We were treated to another torrent of verbal abuse, with the wiring coming in for particular critcism this time. We almost lost a longstanding customer that day, though I believe eventually things were patched up - I expect we wrote off the cost of that display. I'm not sure what the lesson there was - possibly not to take on a completely untried type of display with only two weeks to build it where no known technology exists to make an invisible and transparent connection to 300 LEDs! I have to admit, I wasn't completely unsympathetic to the customer's response to the finished display - it looked like crap.

Modular, modular, modular!

Many, possibly most, of the displays I designed were simple sequential types, where one section would light up after another. While every display was different in its details, after a few of these I realised that electronically they were all nearly identical. I usually implemented these using shift registers with decoded outputs where needed and either a relay or open collector drive output depending on what was needed. The builds were usually on wirewrap. I figured a better way would be to split the units up into modules with a central "controller" that could be programmed using an EPROM with a state-machine type of design (or even a microcontroller for more complex designs), with drive cards plugging in to provide a number of outputs. Need more? Plug in another card. It seemed an obviously good way to do it to me - make a few standard printed circuits which would be much easier to build than wirewrap, and program the displays using a programmed EPROM chip. I even figured I could write a simple application to program the EPROM from a simple description of the display. To my surprise, the boss wouldn't go for it, even as a one-off to try the idea out. I must have done a poor job of selling it to him because he really didn't seem to see any advantages. Oh well, sometimes you just can't get others to see things your way.

Software

A lot of displays we built required software. Over the years the company had used a variety of "platforms" - one popular one had been the venerable Commodore 64. However, by the time I worked there I was pretty well into Macs and had been programming them in Pascal for a while. So that's what we used for the projects I needed custom software for. At the time I used an SE/30 which was fitted with a colour video card with an external 13" monitor with a whopping 640 x 480 pixels! This was actually a decent machine in its day and more than capable for doing the stuff we needed to do. When exhibition displays needed a Mac we'd hire one, usually a IIci. The Navy dockyard display had a Mac-based "race game" software, as did its predecessor featuring a single boat racing against the clock. The software was quite simple - just timing the races and displaying a high score (when someone was on hand to type in competitors' names). The main difference from the average app was the liberal use of colour and huge fonts to give a game-like appearance visible from a distance. The hardware interface to the Mac was using a custom programmed microcontroller which interfaced all sorts of inputs and outputs - switched, optical sensors, relays and LEDs to the serial port. This little interface device was very versatile and made many a job very straightforward.