Maintained list of all remaining issues. Call to help.


I tried to ask in the other thread where are your hands or how to take you by the hand.
I'm not sure if trying to help here I'll be able to help or I'll just waste other people's time.

Hey, I did not forget you :)
The plan is to discuss the whole topic with EvilDragon during our 5 hour trip to the Gamescom.

I personally did not look into it ever and have no idea how to build it.
So what we need is
- A tutorial how to change settings and put them on a boot SD card
- A way to do it somewhat efficiently so it does not take 15 minutes to build the whole thing (no idea if that is an issue)

I don't know enough about the hardware. Not sure if getting some parameters wrong in the DRAM timings could damage the hardware permanently somehow.
If you're desesperate enough, because the hardware you might damage is going to the dust bin anyway, then I guess it should be OK.
If there's some suspicion that the hardware might be permanently damaged by the test, the problem is that then you may not want to test twice on the same board
(because the second test might be the right solution, but not working because the first test damaged the hardware).
And that may be wasteful of boards.
Edit: But worse than waste, you are not sure all your defective boards are equally defective,
or need the same RAM timings, so not trying everything on the same board is a problem too.
Edit2: A compromise could be: make a list beforehand of the test you want to do. Do them all on one board, then on the next board do a different permutation of the same tests.
There's of course a combinatory explosion to take into account, but it's better than nothing, I guess.
The change of order does not need to be random if you have some intuition of which tests are more risky to damage hardware. You could start by the safest.

No worries, there is no risk to permanently damage the PCBs.

As far as I understand, the current situation is that to change DRAM timings one needs to:
0 install the build tooling and git. That might be easier on a similar system, like a Pyra, than on a computer where you need to cross compile.
1 download the U-boot source code for the Pyra. Configure it for building the Pyra boot loader and not for some other machine.
2 edit(*) the source code for MLO (=U-boot SPL = U-Boot 1st stage), Commit or document your changes somehow.
3 recompile the MLO. That may be slower on a Pyra or similar system than in a more powerful computer where you cross compile.
4 install it to a Pyra
5 boot to test. Document your tests.
6 since it didn't work, go back to 2.

(*) The first time you try step 2 you don't edit anything, just build a pristine MLO to test that it works on a workign board, so that you know
you're compiling things right.

How many of those steps do you already know how to do and how many could someone help you do ?
Can you use a Debian system to do it?
Which architecture is your debian system on ?

Basically all I know is how to install git.
It would be super helpful to get a tested README or tutorial of any sorts on how to do the other steps.
We both are experienced with Linux and I am a programmer by profession.
The issue is time. We do not have time to figure it all out by our own.

Cross compiling is a thing, if someone will make a build environment for us. Maybe a Docker image.
Else compiling on a Pyra is fine for me, depending on how long the build takes.

The alternative would be to change MLO so that it loads the setting from some place, and then write a program that is able to write the settings in that place.
The program that writes the settings in that place could be made more or less user friendly relatively easy.
But changing MLO to load the settings from somewhere is difficult, because MLO does not have all the computer ready to work when loading things.
Remember that at that time, it can't even use RAM, because it's not yet initialized, So let alone OS functionality or most libraries. MLO binary code should also be space constrained somehow. You can't write all the code you want because it won't fit in SRAM, cache as RAM or whatever it is. In U-Boot some TPL or SPL stages are able to load the next stage (so then they should be just as able to load RAM settings) and some are not able and have to request it from bootrom or some external system.
So it's not ordinary C programming, although something should be doable.


A third alternative would be some way to edit an MLO binary to just change some constant where the RAM settings are, without needing to recompile.
Could be so easy as: Put some UUID in a constant in the code. Put the RAM settings constant next, put some UUID right after it. Persuade the compiler not to suppress those useless UUIDs that nothing is using.
Use a binary editor to change the constant between the 2 UUIDs. Install the edited binary MLO and test.
But I find that dirty.

I do not think those are options we should consider if not absolutely necessary.
 
Hey, I did not forget you :)
The plan is to discuss the whole topic with EvilDragon during our 5 hour trip to the Gamescom.

Is the Gamescom August 18-22 2024 in Leipzig ?
That's a bit too litle time for trying things before it, I'll try to see what I can do by the 22nd.

I personally did not look into it ever and have no idea how to build it.
So what we need is
- A tutorial how to change settings and put them on a boot SD card
I'll try to build a n MLO. The part of putting it in an SD card I can't test myself, or more precisely, I can't test that the SD card boots, which would be the point.
But on the other hand that should be well documented already.

- A way to do it somewhat efficiently so it does not take 15 minutes to build the whole thing (no idea if that is an issue)
Not sure about that. It depends on the machine where you corss-compile and whether you can chose to compile only MLO or it compiles U-Boot too... We'll see...
But my results on compile time are not going to be comparable to your setup.

No worries, there is no risk to permanently damage the PCBs.
Fine, if that's assured, it makes things easier.
Basically all I know is how to install git.
That's something :)
It would be super helpful to get a tested README or tutorial of any sorts on how to do the other steps.
We both are experienced with Linux and I am a programmer by profession.
The issue is time. We do not have time to figure it all out by our own.
I'll try to do build myself and document dependencies and command line. But there's always some part of trial and error and then is when the experience comes handy. I don't have much in this.
Cross compiling is a thing, if someone will make a build environment for us. Maybe a Docker image.
Will try something. It's going to be a little "works for me" at best, but maybe someone else can improve on it or something...
There's a docker file in the pyraos repo, but if I read it right, it takes the MLO as a binary, it doesn't build it.
I think I'll try a dockerfile from scratch...
Else compiling on a Pyra is fine for me, depending on how long the build takes.
I would guess longer than 15 minutes, I don't know. I could try on a wandboard. But from what you say, you want to cross compile, it's logical.
I do not think those are options we should consider if not absolutely necessary.
OK.
 
Is the Gamescom August 18-22 2024 in Leipzig ?
That's a bit too litle time for trying things before it, I'll try to see what I can do by the 22nd.


Thanks a lot for your help :)

Maybe there is a way to combine your efforts with @Wally. No idea if building the OS is related to the MLO in any way.
 
Uh? Thanks. Then 21st-25th August. OK. I don't know where I got it wrong from...
Thanks a lot for your help :)
Wait until I have anything to offer :) I'm not that master in all this, and I've never player with U-Boot for a board I don't have...
Maybe there is a way to combine your efforts with @Wally. No idea if building the OS is related to the MLO in any way.
Maybe later. The @Wally repo seems to create a bootable image to burn into a SD card. That image includes MLO.
As far as I understand Wally takes the MLO binary as input, it doesn't build it.
It might be expanded, or both @pimaster and @Wally repos used from some new yocto project, or... anything would be possible one day, but let's go step by step.
And the U-Boot we have is 7 years old, unless there's something newer on the Letux side ? At some point it should be updated.
But let's start with basics first.
 
On the topic of LTE connection quality, I would recommend attempting to lock the modem to the frequency you believe is strongest in your area. Do you know if you have a script/interface for this already?

If not, I can dig through old posts about sending AT commands to the modem and attempt to write something that lets you select a band to lock to.
I even made a gui for this and we testet stuff on irc with little luck.
Basically all sensible AT commands have been tried already.
To my unterstanding this is not a frequency problem but an antenna thing, you may search for someting like gain or input switch in the docs, might have missed it, might be my docs where incomplete.
But the thing is the self test command returns that it appears to work just fine, i even can connect to every network, that is , if i am next to the cell tower only tho.
 
I wanted to post this to the wiki, but there are issues there.

If you want help understanding how to do something, you have to tell us what you are having trouble with. Help with everything is really hard.
If you aren't prepared to google some information to help your understanding, then poking around with the boot loader might not be the best idea.

= Compiling Uboot =
== One off tasks. ==
=== Get the source code ===
Most code is hosted on git. You will need git installed.

Code:
git clone `URL`

Some known repositories:
* http://dev.pyra-handheld.com/kernel/pyra-uboot.git/
* http://dev.pyra-handheld.com/daveshah/pyra-uboot.git/
* http://dev.pyra-handheld.com/pimaster/pyra-uboot.git/
Since development on uboot is a bit on the testing/feeling stage, you might have to switch to a specific branch.
Code:
git switch `tag name`
To see branches:
Code:
git branch
=== Get a configuration ===
Uboot can bring up many boards. You need a configuration that is designed to bring up an omap board.

Code:
make omap5_pyra_4gb_defconfig

You can then make minor configuration changes with <code>make menuconfig</code>
Changing ARM architecture > OMAP5 board select from
* Pyra Handheld with LC15 CPU
* Pyra Handheld with OMAP5
This switches between some board code that was written by goldelico to dragonbox. Long term there should be no difference but whilst we are testing, more code gets changed on the dragonbox side.
Want to make sure you're compiling something you can edit? Open `include/configs/omap5_pyra.h` and edit the CONFIG_EXTRA_ENV_SETTINGS. Add an "echo This is my custom uboot" or something creative.
== Compile ==
Easy bit.

Code:
make

Takes about 3 minutes to compile on the Pyra first time. Small changes afterwards should only take 15-30 seconds.

Cross compiling might work with:

Code:
CROSS_COMPILE=arm-linux-gnuabihf make
== Install Uboot ==
The easy way is to copy MLO and u-boot.img to the root of a fat partition. On the Pyra this is good fun because you can put it on an SD card that is in the left slot. If your boot loader doesn't work, you can pop out the SD card and use the boot loader that is installed on the MMC.
 
Regarding u-boot, it is not just the build/compile thing. We also need to know what variables are related to RAM timing.

I guess that is the complicated part.

Takes about 3 minutes to compile on the Pyra first time. Small changes afterwards should only take 15-30 seconds.

That is good to know :)

If you want help understanding how to do something, you have to tell us what you are having trouble with. Help with everything is really hard.
If you aren't prepared to google some information to help your understanding, then poking around with the boot loader might not be the best idea.

Actually I have not looked into this issue at all.
I just wrote down what I remembered from talking to ED some time ago.
If I was not stuck with this absolute mental torture of reworking Pyra housings and having a ton of failing PCBs at hand, which forces me to undo much of the work I did and swap out PCBs from already built Pyras, then I would gladly dive deeper into this issue myself.

Getting some easy instructions on how to build the Uboot and which variables to modify was some wish ED expressed while having a casual talk with me.

I would love to dig deeper into the software issues, however I spent an other 4 days of depressing hardware work during the last week. You have no idea how much of a shit show this all is.
You think that the Pandora situation with TI and 10% failing boards was bad? Well, 12 of the 48 Pyras I built are dead. I will have to open them up again, swap out the PCB and see if it boots now. Guess what, we have 2 PCBs now. Each one of those can have issues or cause a non booting unit. Now imagine how great it feels if you have to repeat that board swapping all over again multiple times just to realize that once the Pyra finally boots a key is stuck and needs some tiny manual rework and an other round of removing the PCB.
 
Regarding u-boot, it is not just the build/compile thing. We also need to know what variables are related to RAM timing.
I thought you knew what you wanted to try.
The RAM init is complex. I don't have the docs nor the hardware to test.
The part to change is apparently the file @pimaster already sent to the other thread. (Gitlab not working now? it worked yesterday)
But you need to understand the hardware to understand what the values there mean and what effects and interactions they have.
The commits in that repo already documented nicely the fields, but you still need to understand the concepts.
C:
    .sdram_config_init              = \
      (3 << 29) /* SDRAM_TYPE: 3 = DDR3, 4 = LPDDR2 */ \
    | (0 << 27) /* IBANK:Internal Bank Position */ \
    | (1 << 24) /* DDR_TERM: DDR3 Termination resistor value */ \
    | (1 << 23) /* DDR2_DDQS: DDR2 Differential DDQS. 1 for compat */ \
    | (0 << 21) /* DYN_ODT: DDR3 Dynamic ODT. Not supported */ \
    | (0 << 20) /* DDR_DISABLE_DLL: Disable DLL */ \
    | (1 << 18) /* SDRAM_DIVE: DDR3, 0 for RZq/6 and 1 for RZQ/7 (DDR_TERM?) */ \
    | (1 << 16) /* CWL: DDR3 CAS Write Latency 0=5, 1=6, 2=7, 3=8 */ \
    | (0 << 14) /* NARROW_MODE: 0=32 bit, 1=16 bit. Only 32 bit supported */ \
    | (6 << 10) /* CL: CAS Latency (RL) DDR2, 3=3clocks, 4=4, 5=5, 6=6, 7=7, 8=8 */ \
        /* DDR3 2=5clocks, 4=6, 6=7, 8=8, 10=9, 12=10, 14=11 */
    | (7 <<  7) /* ROWSIZE: 0=9 through 7=16 row bits */ \
    | (3 <<  4) /* IBANK: Internal banks of SDRam: 0=1, 1=2, 2=4,3=8 */ \
    | (1 <<  3) /* EBANK: External Chip Select Setup 0=NCS0 only. 1=NCS[1:0] */ \
    | (2 <<  0) /* PAGESIZE: 0=256word (8cols), 1=512word (9cols), 2=1024, 3=2048 */,
    .sdram_config                   = \
      (3 << 29) /* SDRAM_TYPE: 3 = DDR3, 4 = LPDDR2 */ \
    | (0 << 27) /* IBANK:Internal Bank Position */ \
    | (1 << 24) /* DDR_TERM: DDR3 Termination resistor value */ \
    | (1 << 23) /* DDR2_DDQS: DDR2 Differential DDQS. 1 for compat */ \
    | (0 << 21) /* DYN_ODT: DDR3 Dynamic ODT. Not supported */ \
    | (0 << 20) /* DDR_DISABLE_DLL: Disable DLL */ \
    | (1 << 18) /* SDRAM_DIVE: DDR3, 0 for RZq/6 and 1 for RZQ/7 (DDR_TERM?) */ \
    | (1 << 16) /* CWL: DDR3 CAS Write Latency 0=5, 1=6, 2=7, 3=8 */ \
    | (0 << 14) /* NARROW_MODE: 0=32 bit, 1=16 bit. Only 32 bit supported */ \
    | (6 << 10) /* CL: CAS Latency (RL) DDR2, 3=3clocks, 4=4, 5=5, 6=6, 7=7, 8=8 */ \
/* DDR3 2=5clocks, 4=6, 6=7, 8=8, 10=9, 12=10, 14=11 */
    | (7 <<  7) /* ROWSIZE: 0=9 through 7=16 row bits */ \
    | (3 <<  4) /* IBANK: Internal banks of SDRam: 0=1, 1=2, 2=4,3=8 */ \
    | (1 <<  3) /* EBANK: External Chip Select Setup 0=NCS0 only. 1=NCS[1:0] */ \
    | (2 <<  0) /* PAGESIZE: 0=256word (8cols), 1=512word (9cols), 2=1024, 3=2048 */,
    .sdram_config2                  = 0x0,
    .ref_ctrl                       = \
      (1 << 31) /* INITREF_DIS: Initialization and Refresh disable. 1=disable */ \
    | (0 << 29) /* SRT: DDR3 Self Refresh temp range. 0=normal*/ \
    | (0 << 28) /* ASR: DDR3 Auto Self Refresh. 1=enable*/ \
    | (0 << 24) /* PASR: Partial Array Self Refresh */ \
    | (0x40F1 << 0) /* REFRESH_RATE: Refresh Rate (< 6*T_RFC) */,
    .ref_ctrl_final                 = \
      (0 << 31) /* INITREF_DIS: Initialization and Refresh disable. 1=disable */ \
    | (0 << 29) /* SRT: DDR3 Self Refresh temp range. 0=normal*/ \
    | (0 << 28) /* ASR: DDR3 Auto Self Refresh. 1=enable*/ \
    | (0 << 24) /* PASR: Partial Array Self Refresh */ \
    | (0x1035 << 0) /* REFRESH_RATE: Refresh Rate (< 6*T_RFC) */,
    .sdram_tim1                     = \
      ((8-1)  << 29) /* 7 T_RTW: between Read to Write data phases.*/ \
    | ((8-1)  << 25) /* 7 T_RP: Precharge to Activate or Refresh. */ \
    | ((8-1)  << 21) /* T_RCD: Activate to Read or Write. */ \
    | ((8-1)  << 17) /* T_WR: last Write transfer to Precharge. */ \
    | ((20-1) << 12) /* T_RAS: Activate to Precharge. >= T_RCD value */ \
    | ((28-1) << 6) /* 27 T_RC: Activate to Activate. */ \
    | ((4-1)  << 3) /* 7 T_RRD: Activate to Activate for a different bank. For an 8-bank, ((tFAW / (4 × tCK)) - 1) */ \
    | ((4-1)  << 0) /* T_WTR: last Write to Read. */,
    .sdram_tim2                     = \
      ((4-1) << 28) /* T_XP: from power-down exit to any command other than a read command */ \
    | (0 << 25) /* T_ODT: from ODT enable to write data driven for DDR3. Must be equal to tAONPD */ \
    | ((281+6-1) << 16) /* T_XSNR: 'T_RFC+6' Self-Refresh exit to any command other than a Read command. For DDR3, the value of tXS must be programmed. */ \
    | ((512-1) << 6) /* T_XSRD: from Self-Refresh exit to a Read command. For DDR3, the value of tXSDLL must be programmed. */ \
    | ((4-1) << 3) /* T_RTP: for the last read command to a Precharge command */ \
    | ((4-1) << 0) /* T_CKE: between CKE pin changes */,
    .sdram_tim3                     = \
      (4 << 28) /* T_PDLL_UL: cycles for PHY DLL to unlock. A value of N will be equal to N x 128 clocks */ \
    | ((3-1) << 24) /* T_CSTA: between write-to-write or read-to-read data phases to different chip selects. */ \
    | ((4+1-1) << 21) /* T_CKESR: 'T_CKE+1' cycles for which SDRAM must remain in Self Refresh */ \
    | ((64-1) << 15) /* ZQ_ZQCS: cycles for a ZQCS command. */ \
    | ((1-1) << 13) /* T_TDQSCKMAX: clock that satisfies tDQSCKmax for LPDDR2. */ \
    | ((281-1) << 4) /* T_RFC: DDR clock cycles from Refresh or Load Mode to Refresh or Activate.*/ \
    | (8 << 0) /* T_RAS_MAX: Maximum number of REFRESH_RATE intervals from Activate to Precharge command */,
    .read_idle_ctrl                 = /* EMIF_DLL_CALIB_CTRL*/ \
      (14 << 16) /* ACK_WAIT: Ack Wait, time after phy_dll_calib */ \
    | (8  << 0 ) /* DLL_CALIB_INTERVAL: Interval between phy_dll_calib generation*/,
    .zq_config                      = \
      (1 << 31) /* ZQ Calibration on CS1 */ \
    | (1 << 30) /* ZQ Calibration on CS0*/ \
    | (0 << 29) /* ZQ Dual Calib. Needs resistor */ \
    | (1 << 28) /* ZQ Self Refresh on exit */ \
    | (0 << 27) /* ZQ Self Refresh on Power Down*/ \
    | ((2-1) << 18) /* Number of ZQCL make ZQINIT (minus 1)*/ \
    | ((4-1) << 16) /* Number of ZQCS intervals that make up ZQCL (minus 1) */ \
    | (0x190B << 0) /* Refresh periods between ZQCS */,
    .temp_alert_config              = 0x00000000,
    .emif_ddr_phy_ctlr_1_init       = \
      (0  << 27) /* RDLVL_MASK: mask read data eye training. Incremental training should be disabled */ \
    | (0  << 26) /* RDLVLGATE_MASK: mask dqs gate training. Incremental training should be disabled */ \
    | (0  << 25) /* WRLVL_MASK: mask write leveling training. Incremental training should be disabled */ \
    | (1  << 21) /* PHY_HALF_DELAYS: 2x mode (MDLL clock is half Phy), 0: 1x mode (MDLL is same as PHY) */ \
    | (1  << 20) /* PHY_CLK_STALL_LEVELEnable variable idle for normal operation.*/ \
    | (0  << 19) /* PHY_DIS_CALIB_RST: Disable dll_calib. Debug only */ \
    | (1  << 18) /* PHY_INVERT_CLK_OUT: Invert DRAM clock. 0 for LDDDR2 */ \
    | (16 << 10) /* PHY_DLL_LOCK_DIFF: Max # delay line taps variation while maintaining the master DLL lock */ \
    | (0  << 9) /* PHY_FAST_DLL_LOCK: Master DLL to lock fast or average logic must be part of locking process*/ \
    | (10 << 0) /* READ_LATENCY: RL + reg_phy_rdc_we_to_re -1.*/,
    .emif_ddr_phy_ctlr_1            = \
      (0  << 27) /* 1: mask read data eye training. Incremental training should be disabled */ \
    | (0  << 26) /* 1: mask dqs gate training. Incremental training should be disabled */ \
    | (0  << 25) /* 1: mask write leveling training. Incremental training should be disabled */ \
    | (1  << 21) /* 1: 2x mode (MDLL clock is half Phy), 0: 1x mode (MDLL is same as PHY) */ \
    | (1  << 20) /* Enable variable idle for normal operation.*/ \
    | (0  << 19) /* Disable dll_calib. Debug only */ \
    | (1  << 18) /* Invert DRAM clock. 0 for LDDDR2 */ \
    | (16 << 10) /* Max # delay line taps variation while maintaining the master DLL lock */ \
    | (0  << 9) /* Master DLL to lock fast or average logic must be part of locking process*/ \
    | (10 << 0) /* READ_LATENCY = RL + reg_phy_rdc_we_to_re -1.*/,
    .emif_ddr_ext_phy_ctrl_1        = \
      (0x040 << 20) /* DLL Ratio Phy 2 */ \
    | (0x100 << 10) /* DLL Ratio Phy 1 */ \
    | (0x100 << 0 ) /* DLL Ratio Phy 0 */,
    .emif_ddr_ext_phy_ctrl_2        = \
      (0x0B7 << 22) /* DQS Phy 1 Rank 0 */ \
    | (0x0B7 << 11) /* DQS Phy 0 Rank 1 */ \
    | (0x0B7 << 0 ) /* DQS Phy 0 Rank 0 */,
    .emif_ddr_ext_phy_ctrl_3        = \
      (0x0B7 << 23) /* DQS Phy 2 Rank 1 */ \
    | (0x0B7 << 12) /* DQS Phy 2 Rank 0 */ \
    | (0x0B7 << 1 ) /* DQS Phy 1 Rank 1 */,
    .emif_ddr_ext_phy_ctrl_4        = \
      (0x0B7 << 13) /* DQS Phy 3 Rank 1 */ \
    | (0x0B7 << 2 ) /* DQS Phy 3 Rank 0 */,
    .emif_ddr_ext_phy_ctrl_5        = 0x4350D435 /*Should be auto filled?*/,
    .emif_rd_wr_lvl_rmp_win         = 0x00000000 /* Not supported */,
    .emif_rd_wr_lvl_rmp_ctl         = (1 << 31) /* Full leveling trigger */,
    .emif_rd_wr_lvl_ctl             = (1 << 31) /* Full leveling trigger */ ,
    .emif_rd_wr_exec_thresh         = /* EMIF_READ_WRITE_EXECUTION_THRESHOLD */ \
      (0 << 31) /* MFLAG_OVERRIDE: MFlag Override*/ \
    | (1 << 30) /* LL_BUBBLE_ENABLE. Allows burst access*/ \
    | ((4-1) << 8) /* WR_THRSH: Number of write bursts (minus 1)*/ \
    | ((6-1) << 0) /* RD_THRSH: Number of read bursts (minus 1)*/

I guess that is the complicated part.
Yes, you have to read the docs for the memory controller on the SOC (EMIF?) and the DRAM chips, and the board layout.
And then understand electronics better than I do. The general idea is that you get dozens of fast lanes between
two chips, (for addresses and for data) and all lanes have to read the value at a single time which is late enough so that all of the bits have stabilised and
show valid values, but soon enough that you don't waste time in a performance critical operation (or you get the data from the next address).
The RAM chips have to read the addresses and data lines and set the data lines strong enough to get to the SOC, fast enough to be on time and without generating interference from one lane to another.
And The SOC has to set addresses and read and set data equally strong, fast and reliably.
The parameters are not only delays and frequencies but usually also electrical properties, like voltage or impedance or so that the chips can regulate to compensate
the printed circuit.
For me it's all black magic.

Getting some easy instructions on how to build the Uboot and which variables to modify was some wish ED expressed while having a casual talk with me.

I would love to dig deeper into the software issues, however I spent an other 4 days of depressing hardware work during the last week. You have no idea how much of a shit show this all is.
You think that the Pandora situation with TI and 10% failing boards was bad? Well, 12 of the 48 Pyras I built are dead. I will have to open them up again, swap out the PCB and see if it boots now. Guess what, we have 2 PCBs now. Each one of those can have issues or cause a non booting unit. Now imagine how great it feels if you have to repeat that board swapping all over again multiple times just to realize that once the Pyra finally boots a key is stuck and needs some tiny manual rework and an other round of removing the PCB.
But you don't have some infrastructure to test boards without mounting a whole pyra ? some mainboard + power supply + serial port connection to a computer or something ?
I thought you only assembled boards into Pyras once the boards were tested ?
You take one known good mainboard and test all CPU boards on it. Then you take one known good CPU board and test it on all mainboards...
And then you mount pyras using PCBs from the drawer of the ones that tested good ?
I think I misunderstand something.

Anyway, thanks for all the work you're putting in.
 
Regarding u-boot, it is not just the build/compile thing. We also need to know what variables are related to RAM timing.

I guess that is the complicated part.



That is good to know :)



Actually I have not looked into this issue at all.
I just wrote down what I remembered from talking to ED some time ago.
If I was not stuck with this absolute mental torture of reworking Pyra housings and having a ton of failing PCBs at hand, which forces me to undo much of the work I did and swap out PCBs from already built Pyras, then I would gladly dive deeper into this issue myself.

Getting some easy instructions on how to build the Uboot and which variables to modify was some wish ED expressed while having a casual talk with me.

I would love to dig deeper into the software issues, however I spent an other 4 days of depressing hardware work during the last week. You have no idea how much of a shit show this all is.
You think that the Pandora situation with TI and 10% failing boards was bad? Well, 12 of the 48 Pyras I built are dead. I will have to open them up again, swap out the PCB and see if it boots now. Guess what, we have 2 PCBs now. Each one of those can have issues or cause a non booting unit. Now imagine how great it feels if you have to repeat that board swapping all over again multiple times just to realize that once the Pyra finally boots a key is stuck and needs some tiny manual rework and an other round of removing the PCB.
This must be extremely frustrating if not infuriating. Thank you for your effort. Don't smash too many pyrii.. that's what I would do.
 
I thought you knew what you wanted to try.
The RAM init is complex. I don't have the docs nor the hardware to test.
The part to change is apparently the file @pimaster already sent to the other thread. (Gitlab not working now? it worked yesterday)
But you need to understand the hardware to understand what the values there mean and what effects and interactions they have.
The commits in that repo already documented nicely the fields, but you still need to understand the concepts.

Since no one is taking care of that issue and we HAVE to solve that issue, we need at some point experiment and do it by ourself.
Like I said, it is was ED who suggested/requested that.

But you don't have some infrastructure to test boards without mounting a whole pyra ? some mainboard + power supply + serial port connection to a computer or something ?
I thought you only assembled boards into Pyras once the boards were tested ?
You take one known good mainboard and test all CPU boards on it. Then you take one known good CPU board and test it on all mainboards...
And then you mount pyras using PCBs from the drawer of the ones that tested good ?
I think I misunderstand something.

ALL of the CPU boards I used ARE tested. Hurray.
I made a test setup just like you suggested.

It also could be the motherboard which causes the boot fail.
One of the parts silently changed their PIN positions slightly which makes it incompatible with ours. Hurray.
It's something Global Components can fix by some reworking but one more bullshit on the way.

This must be extremely frustrating if not infuriating
It is. The worst part is that if I leave, the project will be at a halt once more :(
Any other task I would have just given up.

Producing the upper shells is straight forward now. I can do about 150 a week, all reworking included.
I even figured out how to prepare the shoulder buttons to speed that up a bit. Not by much but a bit.

It's those 2 things that are just stupidly insulting, the shoulder buttons and the failurs.
 
Regarding u-boot, it is not just the build/compile thing. We also need to know what variables are related to RAM timing.

I guess that is the complicated part.
It's hard. I underestimated how hard it was going to be.
The Ti documentation wording does not match the Ram documentation nor do either of them use the same terminology that the DDR spec uses. Sometimes there is something that is close, but it always left me doubting.
And I spoke with a friend who designs circuit boards. He wasn't surprised.

I thought it was going to be a matter of double checking the values. Correcting some and recompiling. I feel like I'm missing something though.
I'm starting to worry that there might be a bring up issue (although the EVM usually works).
One thing I haven't looked into is the length of the RAM lines in relation to some timing configuration. Can't remember the name.

I don't think you can just tinker your way into a solution. You might have to become an expert (which I was looking forward to, but life got in the way)
 
It's hard. I underestimated how hard it was going to be.
The Ti documentation wording does not match the Ram documentation nor do either of them use the same terminology that the DDR spec uses. Sometimes there is something that is close, but it always left me doubting.
And I spoke with a friend who designs circuit boards. He wasn't surprised.

I thought it was going to be a matter of double checking the values. Correcting some and recompiling. I feel like I'm missing something though.
I'm starting to worry that there might be a bring up issue (although the EVM usually works).
One thing I haven't looked into is the length of the RAM lines in relation to some timing configuration. Can't remember the name.

I don't think you can just tinker your way into a solution. You might have to become an expert (which I was looking forward to, but life got in the way)

Regarding EVM: It works all well and good with 2GB of RAM. The 4 GB is the issue.

Regarding tinkering:
ED's approach would be to:
  1. Take a working PCB
  2. Use the current configuration that works with that board
  3. Get a list of all the timing parameters involved
  4. Start with the first one
    1. Lower the value from the working config until it does not boot any longer
      -> Note that down
    2. Increase the value from the working config until it does not boot any longer
      -> Note that down
    3. Now that the middle value by calculating the average
  5. Repeat 4 for every value. Reset all other values to the starting values, so that only one value will deviate from the working start configuration at each time
  6. When all averages are calculated, set all the values to the average we just calculated
  7. Hope that those values will be more stable than the current config.
His reasoning is that by testing the limits for each value, we can determine the range in which each value does work. By setting each one to the middle ground, that would be the best bet for a config that works with each board.

I thought it was going to be a matter of double checking the values. Correcting some and recompiling. I feel like I'm missing something though.

We are on our own here because the the OMAP 5 was never properly finished by TI. They abandoned it. The documentation says that the OMAP will test the timings by itself to the best values. That simply is broken. ED wants to do the average method and hopes to guess a setting that is more compatible than the one we use.

Thanks for consulting your friend and helping us here. We gladly take any helping hand :)

I would suggest to replace the 4GB PCBs with 2GB ones. Maybe ditch the 4GB entirely. I bet people will understand if we have to go that way. There is a huge pile of non booting PCBs here and we'd rather reduce the RAM than to let the project die.
I myself was a strong supporter of the 4GB RAM, however the mess it caused and causes us is beyond belief :(
However, even such a step would not be easy because we still have a lot of 4GB boards and I don't know how much 4GB RAM modules we still have. So going with a 2GB config will also cause use huge costs.

Even if we have to use 2 or 3 RAM timing configs, that would at least save the dead boards. Maybe we can find a config that works with the currently broken PCBs but not the ones that are currently stable. Sure, it would be a little inconvenient for people if they want to flash the OS but it still will be better than the current situation.

I have no idea how much sense that 'average method' makes, however it at least sounds logical.
 
Regarding EVM: It works all well and good with 2GB of RAM. The 4 GB is the issue.
I can believe that, but I suspect from my vast ignorance that it may also not work with different 2GB RAM chips or different PCB layout even with the same 2GB RAM chips.

Regarding tinkering:
ED's approach would be to:
  1. Take a working PCB
  2. Use the current configuration that works with that board
  3. Get a list of all the timing parameters involved
  4. Start with the first one
    1. Lower the value from the working config until it does not boot any longer
      -> Note that down
    2. Increase the value from the working config until it does not boot any longer
      -> Note that down
    3. Now that the middle value by calculating the average
  5. Repeat 4 for every value. Reset all other values to the starting values, so that only one value will deviate from the working start configuration at each time
  6. When all averages are calculated, set all the values to the average we just calculated
  7. Hope that those values will be more stable than the current config.
His reasoning is that by testing the limits for each value, we can determine the range in which each value does work. By setting each one to the middle ground, that would be the best bet for a config that works with each board.
I don't know enough to judge, but I think that should be oversimplistic.
It's almost as if you're thinking in a multidimensional orthogonal space of parameters for lack of a better understanding of the problem, but the real world is never such a nice square grid.
Now, I repeat, I don't understand the problem myself, so I guess your guess is as good as mine.
I'd recommend reading more before trying things so blindly. But I maybe should shut up.Your method might work just because the range you discover for each parameter contains just one value, who knows, but then you'd get what you already have..

We are on our own here because the the OMAP 5 was never properly finished by TI. They abandoned it. The documentation says that the OMAP will test the timings by itself to the best values. That simply is broken. ED wants to do the average method and hopes to guess a setting that is more compatible than the one we use.
What I'd love if possible would be to find what the algorithm should be for testing the parameters (that which the OMAP should do but fails), and code that in MLO.
I have no idea whether the OMAP hardware offers enough knobs to play with to even do that.
It should be done by reading DDR specs, RAM chip specs and programming what's there. @þimaster seems to have already tried to read it and it wasn't easy.
It's likely that implementing the whole autocalibration protocol would be too much code to fit in MLO, though. So it'd be a lot of work for someone more clever than I, but it might not be viable at all.
One might maybe look at other boards in U-Boot using similar RAM with less powerful processors, if any at all, to see if there's some code to be borrowed (but it's unlikely, isn't it?).

Thanks for consulting your friend and helping us here. We gladly take any helping hand :)
Yes, thanks @pimaster, @Askarus and you all.

I would suggest to replace the 4GB PCBs with 2GB ones. Maybe ditch the 4GB entirely. I bet people will understand if we have to go that way. There is a huge pile of non booting PCBs here and we'd rather reduce the RAM than to let the project die.
I myself was a strong supporter of the 4GB RAM, however the mess it caused and causes us is beyond belief :(
However, even such a step would not be easy because we still have a lot of 4GB boards and I don't know how much 4GB RAM modules we still have. So going with a 2GB config will also cause use huge costs.
But I'm not confident that the amount of RAM is the problem. If the PCB layout is different and the RAM chips are different, the RAM settings don't have to be the same.
I'd say if it works with 2GB Pyras, it'd be more chance than logic.
How much experience do we have with 2GB Pyras ? How many were built and tested? Would a production run show the same low yield anyway ?
Even if we have to use 2 or 3 RAM timing configs, that would at least save the dead boards. Maybe we can find a config that works with the currently broken PCBs but not the ones that are currently stable. Sure, it would be a little inconvenient for people if they want to flash the OS but it still will be better than the current situation.
ACK. That would be a lesser evil, but worth the effort. The problem is how to achieve that.
I have no idea how much sense that 'average method' makes, however it at least sounds logical.
I doesn't sound logical to me. This is not some BIOS menu where you input values for a whole lot of BIOS and CPU ROM code to interpret and fine tune into hardware register values.
These are directly register values. They have to be coherent with one another and the hardware, Or I would expect so, I don't know.

It's like saying that you can build furniture by taking the average of each piece dimension and nailing in the middle of it (like build a stool by nailing the four legs in the middle of the seat).

Apart from that, the RAM timings have a huge impact on system performance. You always want the fastest that is rock stable, never some average.
Having the system boot is only the first part of the problem, it needs to stay working, and do so at decent speeds, even if not all boards are made to work equally fast.

Also, in my ignorant imagination if you take average values you could find problems like one side sampling data when the other side has already sent the next word, or so on.

Some parameters are clearly not averageable (version of DDR? number of banks to use?). So you still need a minimum of understanding to use any average tactic.

But don't let me distract you. I'll try to shut up and play with docker in between housework. I've discovered podman. Sounds a good idea, but I guess I'll stick to docker for now.
 
I would suggest to replace the 4GB PCBs with 2GB ones. Maybe ditch the 4GB entirely.

But I'm not confident that the amount of RAM is the problem.
Stupid random thoughts. Have we tried the 4GB models with 2GB settings? I wonder if there is some bring up code that does something different with 4GB or PAE enabled. That would be a kick in the pants.
It could be a power issue, but I don't know how to check for such a minor fluctuation.
The OMAP5432 has to work with 2 ram chips. It gets a high 16bits from one and a low 16bits from another. I remember reading that in the documentation.
I thought the difference between the 2GB and 4GB was drop in replacement chips. I don't think it is because we have 2 vs 4 chips on the board.
Disabling a pair of chips might be an interesting test on its own.
 
Back
Top