Share your projects


So data caches are kinda trendy these days and I always thought most were crap because they use TCP and do all that copying. Wouldn't it be faster and cooler if it just used shared memory? Then I realised I could do it in a decentralized, serverless way if I use compare-and-swap atomic cpu operations and memory barriers. Just got rid of the last race condition my stress test revealed so this is nearing completion. It's called shampoo.

Shampoo has two data structures, a hash and a circular heap. This is how it hangs together:
1725723971224.png


A lot of very fiddly (nasty) pointer programming. One of the most complicated things I have ever programmed. I'm glad pointer arithmetic is considered unsafe these days. It is.


Comments appreciated :)
 
Last edited:
so "head" points to the next free memory location in the heap. if you want to reserve 50 bytes you need to update head=head+50 so you get those bytes and no one else takes them. i use compare-and-swap so if head gets updated concurrently it's detected and you can try to grab the next 50. it's shared data with no locking or central process. simpler = faster. that's it. if the pointer arithmetic is ever wrong it's usually a jump to some random invalide memory segment (aka segfault) or just hangs and it's usually not worth diagnosing it's too low level so i just keep undoing changes until it's back to normal.
 
Last edited:
Don't be like that. They're just variables, who's values are addresses in memory.
Addresses, huh?
*pumps his shotgun*
Dem folks're lucky I dunno where dey live.
*spits, pumps his shotgun again*

Nah. I did fiddle around with em in cpp and still don't quite know why I inflicted it on myself. I did get to a place where I could work with it, but it was fairly basic stuff. Guess I've not run into the need to manually manage memory yet. Using something like Godot is just too convenient for the silly little things I do.
 
Last edited:
@kuru Often it's way more efficient to communicate where in memory something is than to copy said something (at every step).
The error-prone tasks when dealing with pointers are
- not to fuck up one's pointer arithmetic
- always have a clear concept of pointer (or rather memory (per allocation)) ownership in mind
in cpp std::shared_ptr/std::weak_ptr and std::unique_ptr help with the latter (steer clear of raw c pointers whereever possible (or be very mindful))
 
A long time ago I implemented a void pointer array in C for an embedded OS, which was a hassle.
While I like you knew what was happening and solving a technical challenge, the modern programming languages make life much easier.
 
Let me give another example of pointer shenanigans, in C. To explicitly control memory layout you can cast a certain region of memory to an object or struct.

byte[] memory = byte[1024]; // allocate memory
byte* mem_ptr = memory; // acquire pointer to memory
MyObj *obj1_ptr = (MyObj*) mem_ptr; // use the beginning memory for obj1
MyObj *obj2_ptr = (MyObj*) mem_ptr + 36; // use later bytes for subsequent objects
MyObj *obj3_ptr = (MyObj*) mem_ptr + 72;


Imagine if the offset (36, 72) is wrong. What happens? You get overlapping objects, misinterpreted data aka random crashes.

Now this is a trivial example, in shampoo the objects all have different lengths.


So pointer arithmetic is simple math but when you get it wrong there usually aren't friendly error messages. That's it!

I think this kind of programming - making the most of resources via low level constructs - is common in video games actually.

I should have been a video game programmer ..
 
Was rewriting a java graphical environment for students to python because they decided the whole program should be switched. So I made a pySDL2 version, packaged it and now made a breakout clone in it. This is my first full game including score, game over message some bonusses etc. So yeah the graphical environment works :P
 
The disk on my g4 cube died so i'm rebuilding it from scratch. I'm quite pleased with the apps installed so far. I didn't use it for gaming at all because I had a Nintendo, so it's hard to find reasons to use it today. I think I've got most of the most nostalgic apps installed. AfterDark not shown ^^

Picture 2.png

Post automatically merged:

1000009948.jpg
 
I also run MacOS 7.5 on MiSTer, it's a throwback. Legend is originally MacPaint/Draw/Write were considered so good 3rd party developers didn't know what to do so Apple spun the apps off as Claris.
 
I have a mac mini a 2012 possibly dont remember. It holds the door of my PC case open so the fans work better...lol
The Cube doesn't have a fan. But it was the precursor to the Mini. It's what Steve Jobs wanted on his desk. Hence no fan, vector unit, design masterpiece, etc.
 
At least once per year, I have to reverse-engineer a binary format. I just like tinkering with it and the detective work needed for it. Fortunately, there is always someone who needs this and has no time, software or skills to do it. So a friend got me the files, however, he warned that this time it will be the hard one - OK, if it's not encrypted. The thing is that it was not so much hard, but tedious and just outwards sick. I was thinking that I could explain my ramblings by making interoperability with some CADs (as in EU to do it legally it must be for interoperability), screw this, I had to do it to maintain interoperability with the euclidean 3D universe, general causality and the leftovers of sanity I saved after analyzing these files.
This time I got a strange thing in my hands - a closed and proprietary format of computer simulations (and it's not my first contact with FEA) for a very hermetic, expensive and futuristic software for process design, optimization, manufacturing and organizing. The thing is developed for 35 years and probably some parts have never been modified. I could not get the software, but files - yes, in any amount, with process and result visualizations. With this and the hex editor, I started to chew thru the binaries.
These files are usually used as input for solver, the big program which calculates a lot and its binary input is usually some simple mutation of Fortran or C-like data structures for a simple reason - programmers have no time to write complex things for the I/O, it's more important to solve this 400k x 400k sparse matrices problem than to pack integers and floats. It is linearly slurped at the beginning and linearly spewed out at the end. There is a set of meshes, described usually by vertices and links between, the description of which side goes where, and some sets of values (BCs or results) in points, sometimes more points than vertices (so they are centers of the mesh elements then), or surfaces, or 3D. Simple. Sometimes the method of packing data to bytes may be strange, but it's usually not a terrible thing, many times being a result of some past hardware dependency.

You know this old saying, that who works in Vienna sausage factory never eats Vienna sausages. Now I'm out when it comes to using this "software".

What can we expect? Encrypted files? No. Maybe carefully crafted data structures for describing specific things, and then alternate way of describing it, being a quick memory barf, like the big graphics company starting with letter A did numerous times? No! This is even worse. The terrible soup of text and binary, of data and executable commands, of multiple definition of blocks... Of units definitely not fitting any engineering software in this Earth... This is not nuts, this is a thing straight from some nightmare:
- Time is usually stored in seconds. Entire process time is in units about 30.1M seconds. There is nothing in between.
- In dimensions, the main Scalar unit can be safely approximated as 0.9mm, but the way it is stored leaves some bit garbage in last places after comma. Program translates everything to mm, trimming these bits.
- Then, there is THE unit used in the 3D coordinates, computed who-knows-how, but stored in a control block once and used since then per computation. This is a 3D block having VxWxX Scalar units counted righthandly. All 3D sizes, coordinates and such are written using THE unit in floats.
- If anything has to be written as a 2D surface, it is written using only W units, floats only. 3D Bounding boxes for fields are considered as 2D. Cross-sections are considered still 3D.
- Distances are in Scalar units. And this ends floats in dimensions.
- Result floats are scaled from integers using pre-defined ranges.
- The 4-byte signed integer is the holly binary unit, used nearly everywhere and for nearly everything except dimensions and time. If something does not fit to it, it's zero-padded.
Now the file itself. Some files had proper magic numbers, some not. These which had no numbers, had their versions detected somehow, and I found how by throwing e-mails with person having software: Program determines version by file's modification date. Of many don't-s in programming this seems to be the worst thing as there are so many events in which this gets messed up.
The file contains extensive amount of metadata, including the "cryptographically-secure" storage of user-sensitive information in strings: Lengths are stored like C# compressed uints, but it's the first, not the last bit for continuing and... the strings are written backwards :D. Network data for MPI are "encrypted" by stuffing \0s between each byte, this time without reversing. Absolute paths are everywhere and it's UI's job to figure where is what. Paths are written in mesh names, and it's the UI job to separate this.
To write the surface distribution of values, the mesh gets UV mapping and something surprisingly similar to multi-channel TGA file with incomplete header. One channel-one scalar. I don't know who invented this, but gets the McGyver's award for the most lazy description of 3D mesh surfrace. On the other hand, I could make a C++ program which allows to paint surface conditions in GIMP :D. No preprocessor has this feature!
If the user is tempted to name the file specific way, the version gets upgraded and downgraded specifically. Found by accident.
And there is a "extract" block, as I called it. Any binary or text structure can be in this block and this will be happily extracted to simulation directory when the file is processed by solver. If the extracted file has specific name, the contents will be downloaded without question. I made files which successfully downloaded files from other directories, SMB shares, FTP servers and HTTP sites, the only thing I not tested was Gemini and Gopher. Well, this could be a smoking-gun evidence of a backdoor, but it is actively used in multi-machine computation to obtain missing data from another computer.

Finally, the file has a checksum! Not that it's checked, only shown in program's log at the end of loading so I could craft a file with borked checksum and it was still operable, but I just had to determine how it's calculated. I found that it depends on certain sizes and numbers of structures, but there was always something else. Only adding totally blank data records could increase the checksum by one or two, depending on number of headers it had to write. Adding a comment to the file, for example, could change the checksum, but could leave it untouched, and I found that it mostly depends on the length of the comment's starting, not the comment itself. Finally, the person having the program sent me the report of program's crash (crafting the file with 1500 empty meshes was a bad idea), in which I found that the file is loaded using a 256-ints buffer. Now everything clicked in - the checksum contained the sum of it! The structure read from file in a specific step is not especially 256-ints long, and the buffer is never cleared, only the \0 is set. It means that loading/saving file is not a procedure, but a ritual which, when done wrong, will result in properly loaded file and improperly calculated checksum.

I wonder what was thinking of the developer who implemented this. Some of these things could be results of per-architecture optimizations (data lengths), some of them for some kind of computation method maybe (integer scaling of floats), but the checksum to verify the methodology of loading a file? Non axis-uniform unit system in which the Knuth's Potrzebie System of Measures is pleasant to use? Timing processes taking seconds in units a bit smaller than year?
 
Last edited:
At least once per year, I have to reverse-engineer a binary format. I just like tinkering with it and the detective work needed for it.
I hope AI can help with stuff like this. There are many legacy systems using weird file formats. It would also be helpful to have the application available instead of only the generated files.
 
Usually there are some insider things which must be deduced before, and I tried AI in this - usually it ends with saying that this is some in-house format. So the research includes not only the files, but also who could make it, when and in which machine.
A few years ago I was reversing the format of a simulation software in which results were written in 4-record structures. To get the result, it was needed to perform a very specific join on these records, always ignoring the fourth one. When I was talking with developers of the software, I asked about it. Instead of standard "this file is not readable by user", they finally explained me why 4 records: When they developed the format in late 1980s, the best machine they had was a 4-CPU Sun computer. 3 CPUs doing calculations, one doing visualization, so the 4th record is always skipped. After they switched to MPI with parallel computations, they decided they will abandon the 4-record format in a few years, but they lost the source of format I/O library - only the 32-bit binary object survived and they still use it.
The funny thing is, recently they made their own Python module to read these files, and they reversed the binary object instead of files, relying on edge cases instead of typical file structure. As a result, their own code has bugs and the program does not read some parts.
 
Back
Top