What is rakestrumming?

    The rise of Guitar Hero custom charts led to competitive players discovering new methods to hit challenging sections. One of the biggest breakthroughs, however, is the technique called "rakestrumming". I believe the term was named after players realized that you can send many more strum inputs than you normally would, by spreading out your fingers in the shape of a rake, and then completing the strum motion several times with each finger. If I were to spread out my middle finger, index finger, and thumb and flick my strummer in one clean motion with all those fingers, I can register three inputs in one motion. However, those with Les Paul controllers discovered that you don't even need to use multiple fingers to send additional inputs. Simply flicking the strummer with enough speed with one finger would cause the strummer to start oscillating between the up and down strum switches. With this mechanical ringing of the strummer, some players have been able to make 2/2, 3/3, and sometimes even 4/4 guitars (where X/Y guitar means X strums on the downrake, and Y strums on the uprake.)

Here is a video of me, using rakestrumming to FC the fast strumming section of Soulless 6, an April fool's chart that was created with the sole purpose of being impossible. This particular section peaks at around 48 strums per second, and I used rakestrumming to demonstrate rakestrumming's versatility.

What's the point of this article?

    Since the motion of the strummer during a rakestrum is inherently an underdamped impulse response dictated by the spring, mass, and damping coefficients of the whole strummer assembly, it's actually surprisingly difficult to modify how many strums you get with an up or down rake motion. What do I do if I want to play Crisis City, and then play Soulless 6 later? I can't just manifest a 2/2 and 3/4 guitar out of thin air. Some players have actually had luck changing the rake of their guitars by drowning the strumbar in WD-40, inserting paper behind the strummer PCB, or putting tape between the strummer and the front plate. However, after I had been exposed to hall-effect rapid-trigger switches by one of my college friends, I had a feeling that the technology could make the rakestum not only custom, but also consistent.

Hardware

    For my hall effect switch setup, I didn't buy hall effect keyboard switches. I instead bought these 49E hall effect sensors, and these magnets. To mount them, I first reverse engineered this strummer in FreeCAD so I could insert zero-tolerance snap fits for the magnets, and then I designed zero-tolerance snap fits for the hall effect sensors. The hall effect sensor mounts to the screws on the strummer PCB. I 3D printed the new strummer and mounts, and assembled everything together, with a little help of superglue and tape to ensure the naked hall effect sensors aren't shorting, and that the tiny hall effect sensors don't slip out of their mounts since their wires are always "pushing" them around due to the solid-core tension. I had to fix a few solder joints early on because I was getting some crazy noise on the signals, but at this point, it appears to be sensing perfectly.

Pictures of the setup

Early testing

    I thought this was going to be a piece of cake, the first and second derivatives of the hall effect signal will tell us everything! But nope, I noticed that the acceleration (second derivative) graph is too noisy, and even after denoising, the signal isn't too useful in classifying strums vs. standard strums. Velocity (first derivative) looks promising, but really fast strumming where you are practically vibrating the strummer, looks very similar to super light and quick rakestrums. Finally, I threw out the derivatives, and just settled on the denoised sensor output. Indeed, I can see some notable oscillation on rakes vs. standard strums when looking at the hall effect output.

A1 & A2 sensor value vs. sample during normal tap

A1 & A2 sensor value vs. sample during rakestrum

    At first, I checked if there was a second peak at a sufficient magnitude on the falling edge of the strum motion. This seemed to create a lot of false positives when strumming fast enough that the user isn't fully completing a strum motion, but is close to it. Second, I simply checked to see if the initial strum motion occurred in a short duration, but this of course ended up missing rakes, and inserting unwanted rakes, between quick taps of the strummer registering a rake, and some particularly hard rakes ringing for an unexpectedly long duration. Ultimately, the best luck I had with this setup was the latter, and was able to hit the Soulless 6 strumming verse mentioned earlier.

Stage five escalation

    As my dad likes to say when we're working on cars, "Alright, looks like we need to bring out stage five escalation." When working with cars, it's indeed quite an escalation, involving buying harbor freight engine hoists to pull engines when a bearing is spun, or using an angle grinder to cut off a brake caliper housing when a bolt gets stripped. In this case, I'm going to throw machine learning at the strummer signal to classify the different types of strums. Maybe it's stage five escalation, I'm not sure, the machine learning stuff nowadays is pretty commonplace.

Ahem...

    Anyways, I knew that this is a problem that machine learning could solve. I took a graduate class on machine learning during my undergrad, and actually quite enjoyed it even though it took place after the "AI is stealing the jobs of artists" fiasco, etc. I still have my moral reservations about specifically the term artificial intelligence, but after learning the fundamentals of machine learning in school, I found that it's actually a very elaborate computing technique of numerically finding out what goes in the black box of a function if you only know its inputs and outputs. In other words, I can't seem to find what goes in my "rake vs. standard strum classification function"... Let's let a neural network figure it out!

    I started with perhaps the most important part of any machine learning task, and that is collecting lots of clean data. I sat in my room for about two hours, just strumming away on my controller, classifying the clean results as rakestrums vs. normal strums. I tried to make sure to encapsulate every motion, including slow rakes, fast rakes, quickly releasing a held strum (to traditionally induce a rake,) slow strumming, tapping the strummer, fast strumming, vibrate strumming, the whole deal. After about 100 strum captures for each hall effect sensor, for each classification of strum, I started playing with neural network configurations to find something that is efficient, and accurate.

First model

    I settled on using the filtered hall effect sensor output as my model inputs, and collected 56 ms of sensor outputs, inputting the first 8 ms of samples before the strum release, and the 48 ms of samples after. After training, I was actually really surprised to see the results.

First sensor confusion matrix

Second sensor confusion matrix

First model analysis

    For those uninitiated, a confusion matrix shows the correctly predicted strums, as well as the false positives, and false negatives. The first model actually blew my mind, since the model only had one layer of four dense neurons, followed by a singular sigmoid neuron layer for the boolean classification of "yes, it raked" vs. "no, it didn't rake". Despite this small model, it only missed one rake on the first sensor, and accidentally raked twice on the second sensor. Of course, I would prefer the model to miss rakes rather than accidentally insert them, but oh well. I rushed to put this onto the MCU and test it out.

First mode performance

    The performance... left a little to be desired. Before I had it sending game inputs, I had it simply printing if it rakestrummed vs. normal strummed, and it actually felt magical, that it was this accurate. Once I had it sending inputs however, I noticed that it performed poorly. Most notably, there was, of course, some latency for the machine learning model to classify the type of strum. In this case, it was 15ms of latency. Not too bad, but keep in mind, it applies the 15ms after the 56 ms of strum samples. I could get it to rake well, but I had to fret notes really late and strum really early, which felt really awkward. Secondly, I discovered that playing the game makes you send much different inputs than just aimlessly strumming with no game (for example, when generating training data.) Whenever I would vibrate strum or otherwise strum pretty fast and hard, it would send rakes when I don't want it to. Clearly, I needed to reduce the latency, and also add some more training data for vibrate strumming.

Second model

    Remember how I took the first derivative of the sensor data early on? Well, I kept that data for my training set, so I decided to play with how early the model starts seeing information, and how useful the velocity information is. Turns out, the velocity information works spectacularly well for the model. Especially when the model collects velocity information 40ms before the strum release, and 16ms after. Do you see the unintentional side effect that this creates that I might've doctored to be an intentional side effect? That's right, if it starts the model earlier in the strum motion, it can get to the classification result sooner, thus reducing the latency!

First sensor confusion matrix

Second sensor confusion matrix

Second model analysis

The second model does still misclassify strums, but the first sensor was actually perfect! For those wondering, my data is a 30/20/50 split of training, test, and validation, respectively. Out of curiosity, I made the training spit out the strum packets that it was confidently wrong on identifying to see if I had just given it bad data.

Misclassifications

I'll give the computer some credit. These packets are on the harder-to-classify scale, but it is still definitely wrong. Let's see the performance...

Second model performance

    It's... perfect... I actually cannot believe how well it works. It feels like it's magical, like an intelligent being is watching me play, and sending inputs how I want them... oh wait. Of course the model works well, but it also solved another problem I was aiming to solve. When I introduced the magnets into the strummer and used the default mechanical switches, it actually turned my guitar into a perfect 2/2 rakstrum. It was awesome, except for one tiny detail. It was a little too sensitive, and would rakestrum if I simply let off the strummer too quickly. However, now that I trained a machine learning model on knowing to not do that, the issue has resolved. Again, it feels crazy, since I can use the strummer really loosely, but yet it knows exactly what I intended to do.

Conclusion

     Well, I don't really have an elaborate conclusion apart from "yay, I have a super performant Guitar Hero controller." So in that case, here are some sections I hit using the new setup:

• TTFAF strumming section, showing the seamless transition between raking and standard strums with the guitar as a 2/2, with no overstrum
• Fatalism D from Soulless 4++, showing no overstrums from letting off strummer too fast. Notice the ringing you hear after green strums, but yet no accidental rake
• Solo 5E from Highway to Oblivion Overchart, which uses both standard strumming and rakestrumming in the same technical section