User Tools

Site Tools


Differences

This shows you the differences between two versions of the page.

Link to this comparison view

playground:mission-log-template [2016/03/18 11:18]
chrono [Step 1: Preparation]
playground:mission-log-template [2017/07/08 11:09] (current)
chrono
Line 1: Line 1:
 ~~DRAFT~~ ~~DRAFT~~
-====== ​Harder Soft-Unbricking a Ubiquiti Unifi UAP-Pro AP======+====== ​GHETTO-RAID Voodoo ​======
  
-Image sideview+https://​www.amazon.de/​gp/​product/​B01H43DNPY/​ 
 +http://​www.conrad.com/​ce/​en/​product/​1420906/​RAID-controller-card-2x-M2-NGFF-1x-SATA-socket-2-pin
  
-After re-deploying the Ubiquiti Unifi UAP-Pro at the new base-camp, it simply didn't work anymore. No DHCP reaction, nothing. Except for the LED blinking white sometimes, it seemed completely broken. After opening and attaching a terminal to the serial console, the boot output didn't look promising but at least it wasn't completely dead. So we had to get our hands dirty, pop the hood and hack around to fix this jffs2 issue and revive the AP without costly and probably long/​painful RMA. 
  
-===== The Symptom =====+Manual:
  
-<code+<blockquote
-U-Boot unifi-v1.2.3.74-ge597862a (Mar 14 2013 - 18:30:11) +  Install M.2 NGFF SATA SSD into M2_1 and M2_2 sockets
- +  Set J1 RAID Jumper to your desired mode or RAID initial mode
-Board: Ubiquiti Networks AR9344 board (e507-27.2122.0030.0030) +  ​- Set J2 jumper ​to short SET – GND
-DRAM:  128 MB +  Connect 22-Pin SATA connector to SATA Host and Power
-Flash: 16 MB +  Turn on the power
-Net:   ​eth0 +  Check “Device Manager” the RAID status of M.2 SSD 
-Hit any key to stop autoboot: ​ 0 +  - Set J2 jumper ​to short SET – NC and disable RAID Mode Change 
-Scanning JFFS2 FS: . +<​cite>​[[http://www.produktinfo.conrad.com/datenblaetter/1400000-1499999/001420906-an-01-ml-H_W_RAID_KONTROLLER_2X_de_en_fr_nl_it_pl.pdf|User Manual]]</cite
-Unknown node type: e008 len 52 offset 0xf41eec +</blockquote>
-[...]                                                                                                                                       ​done. +
-cant find .firewall.uci-mWgTyx�� +
-[...] +
-cant find .wireless.uci-HxpWh9 +
-cant find .network.uci-anmaB1 +
-����������������������������������������������������������������������� +
-����������������������������������������������������������������������� +
-����������������������������������������������������������������������� +
-������������������������������������������������������� +
-</​code>​ +
- +
-It is still unclear what exactly causes this jffs2 corruption, but there have been at least three other reported cases and one directly confirmed by djs500, who found the draft-wip-template for this mission log on the net, had exactly the same problem and came back with positive feedback that he was able to fix it using Solution 3, as we did too+
- +
-===== Solution 1 ===== +
- +
-https://​community.ubnt.com/​t5/​UniFi-Troubleshooting/​UniFi-TFTP-soft-recovery-for-bricked-access-point/​ta-p/​607605 +
- +
- +
-Set a static IP on your PC's NIC from 192.168.1.0/24 range, but not 192.168.1.20 (this is the Unifi AP default TFTP IP). +
-  +
-Follow the steps to unbrick your UAP: +
-  - Set the IP on your PC to be able to access the bricked unit and prepare the firmware file. +
-  - On your PC, open your TFTP client and locate the firmware.bin file so you can start it later. Do not initiate the transfer. +
-Using the Windows integrated TFTP client the command to prepare is"tftp -i 192.168.1.20 PUT [path to file]\firmware.bin"​ +
-Unplug the bricked unit. +
-Plug the LAN connection of the PoE injector directly to your PC's NIC. +
-Keep the UniFi AP's reset button depressed and plug in network/PoE in the unit. +
-Keep the reset button depressed until you see the light cycling relatively fast through amber/green/off colors (~14 seconds from power on) -> Release it. Now the device is in TFTP transfer mode. +
-Engage the TFTP push command and wait. The device will write the firmware and it will reboot. +
-If you wait too long to start the TFTP transfer, the push will not work as the device will stall. Please redo from step 3. +
-To ensure all goes clean, after the device gets stable (blinking or steady amber LED), give it a reset: +
-Remove power. Reconnect holding the reset button for ~7 seconds (green led will flash once) -> release reset button and wait for the device to stabilize again. +
-After device is stabilized, power cycle it again, and you are good to go on adopting the device in the controller again. +
- +
-<WRAP round alert> +
-<​html><​p style="​line-height: 40px; font-size: 26px;">​ +
-Official UBNT solution didn't work at all +
-</p+
-</html +
-</​WRAP>​ +
- +
-This may be due to the fact that it expects a bricked UBNT firmware and NOT an OpenWRT installation. +
-===== Solution 2 ===== +
- +
-Soft Unbrick via Serial using urescue and a tftp server was another method mentioned somewhere, but sadly it never really flashed anything and went staight to the reset in less than a second instead. It could never have written the complete firmware in this short time period and after testing: it didn'​t,​ so "​Firmware update complete"​ was a lie:+
  
 <​code>​ <​code>​
-ar7240> urescue +[38198.548757] ata6: exception Emask 0x10 SAct 0x0 SErr 0x4090000 action 0xe frozen 
-Setting default IP 192.168.1.20 +[38198.548761] ata6: irq_stat 0x00400040, connection status changed 
-Starting TFTP server... +[38198.548765] ata6: SError: { PHYRdyChg 10B8B DevExch } 
-Using eth0 (192.168.1.20), address0x81000000 +[38198.548774] ata6: hard resetting link 
-Waiting for connection: - +[38204.283034] ata6: link is slow to respond, please be patient (ready=0) 
-Receiving file from 192.168.1.254:39067 +[38208.562937] ata6: COMRESET failed (errno=-16) 
-Received 4683453 bytes +[38208.562943] ata6: hard resetting link 
-Firmware VersionBZ.ar934x.v3.2.1.2601.140606.1622 +[38214.346804] ata6link is slow to respond, please be patient (ready=0) 
-Setting U-Boot environment variables +[38218.594711] ata6COMRESET failed (errno=-16) 
-Will not overwrite u-boot partition! Skipped.+[38218.594718] ata6: hard resetting link 
 +[38224.362575] ata6: link is slow to respond, please be patient (ready=0) 
 +[38253.649908] ata6COMRESET failed (errno=-16) 
 +[38253.649916] ata6limiting SATA link speed to 3.0 Gbps 
 +[38253.649919] ata6: hard resetting link 
 +[38258.665790] ata6: COMRESET failed (errno=-16) 
 +[38258.665797] ata6: reset failed, giving up 
 +[38258.665803] ata6: EH complete
  
-Firmware update complete. +[45419.920134] ​ sdb: sdb1 sdb2 sdb3 sdb4
- +
-Resetting..+
 </​code>​ </​code>​
  
-<WRAP round alert> +In order to reset it was necessary ​to power offJ2 to SET-GNDJ1 to PMpower onwait couple ​of mins, power off, set J1 to R1 (Mirror), leave J2 on SET-GNDpower on
-<​html><​p style="​line-height:​ 40px; font-size: 26px;">​ +
-Serial urescue didn't work at all +
-</​p>​ +
-</​html>​  +
-</​WRAP>​ +
- +
-===== Solution 3 ===== +
- +
-None of the above solutions worked with our OpenWRT UAP-Pro, so we had to grab one of the original jffs2 images from a live one with original UBNT firmware and use a tftp server to transfer that jffs2 image file to the AP and write the image by handwhich finally worked.  +
- +
-The following unbrick trace documents the 9 easy to follow steps codecfpletz and I took to restore the UAP to its orginal UBNT firmware with default configuration (and default ubnt login) to be able to finally reinstall OpenWRT. +
- +
- +
-==== Step 1: Preparation ==== +
- +
-<WRAP round download>​ +
-**If you've lost your original firmware backup tooyou can download the jffs2 image here: **\\ +
-{{https://​apollo.open-resource.org/​downloads/​jffs2.img.bz2|}}\\ +
-</​WRAP>​ +
- +
-Just bunzip2 the downloaded jffs2.img.bz2 and put jffs2.img into your tftproot so that the AP can access it via LAN. If you don't know which tftp-server to pickdnsmasq and atftp have been reliable alternatives and have solid base of documentation and HOWTOs ​to get started out there. +
- +
-In this setup dnsmasq with tftp enabled was hosted ​on a laptop, eth0 IP 192.168.1.254 connected via LAN to the main port of the UAP-Pro. +
- +
-==== Step 2: Connect Serial Console ==== +
- +
-image: ttl-serial-connection-info +
- +
-==== Step 3: Transfer jffs2 image to AP ==== +
- +
-Power up the AP and press a key to enter u-boot menuthen transfer and load the original +
-jffs2 image over tftp to the AP and store it in RAMstarting at memory address **0x83000000**:​+
  
 <​code>​ <​code>​
-ar7240> tftp 83000000 jffs2.img +[44010.027895] ata6: exception Emask 0x50 SAct 0x0 SErr 0x40d0800 action 0xe frozen 
-</​code>​ +[44010.027899] ata6: irq_stat 0x00400040connection status changed 
- +[44010.027904] ata6: SError: { HostInt PHYRdyChg CommWake 10B8B DevExch } 
-If the tftp server is reachablethe output should look like this: +[44010.027912] ata6hard resetting link 
- +[44010.757949] ata6SATA link down (SStatus 0 SControl 300
-<​code>​ +[44010.757963] ata6EH complete 
-Using eth0 device +[44021.865962] ata6exception Emask 0x10 SAct 0x0 SErr 0x40d0002 action 0xe frozen 
-TFTP from server 192.168.1.254; our IP address is 192.168.1.20 +[44021.865967] ata6irq_stat 0x00000040, connection status changed 
-Filename 'jffs2.img'​. +[44021.865971] ata6SError{ RecovComm PHYRdyChg CommWake 10B8B DevExch } 
-Load address0x83000000 +[44021.865980] ata6hard resetting link 
-Loading#################################################################​ +[44023.085667] ata6SATA link up 6.Gbps (SStatus 133 SControl 300
-         #################################################################​ +[44023.085944] ata6.00ATA-7JMicron H/W RAID132201020max UDMA/133 
-         #################################################################​ +[44023.085948] ata6.00: 1953431552 ​sectors, multi 1: LBA48 
-         #################################################################​ +[44023.086236] ata6.00configured for UDMA/133 
-         #################################################################​ +[44023.086247] ata6: EH complete 
-         #################################################################​ +[44023.086563] scsi 5:0:0:0: Direct-Access ​    ​ATA ​     JMicron H/W RAID 1020 PQ0 ANSI: 5 
-         #################################################################​ +[44023.112971] sd 5:0:0:0: Attached scsi generic sg2 type 0 
-         #################################################################​ +[44023.113113] sd 5:0:0:0: [sdb] 1953431552 512-byte logical blocks: ​(1.00 TB/931 GiB) 
-         #################################################################​ +[44023.113140] sd 5:0:0:0[sdb] Write Protect is off 
-         #################################################################​ +[44023.113144] sd 5:0:0:0: [sdb] Mode Sense: 00 3a 00 00 
-         #################################################################​ +[44023.113200] sd 5:0:0:0[sdb] Write cachedisabled, read cache: enabled, doesn'​t support DPO or FUA 
-         #################################################################​ +[44023.114644] sd 5:0:0:0: [sdb] Attached SCSI disk
-         #################################################################​ +
-         #################################################################​ +
-         #################################################################​ +
-         #################################################################​ +
-         #################################################################​ +
-         #################################################################​ +
-         #################################################################​ +
-         #################################################################​ +
-         #################################################################​ +
-         #################################################################​ +
-         #################################################################​ +
-         #################################################################​ +
-         #################################################################​ +
-         #################################################################​ +
-         #################################################################​ +
-         #################################################################​ +
-         #################################################################​ +
-         #################################################################​ +
-         #################################################################​ +
-         #################################################################​ +
-         #################################################################​ +
-         #################################################################​ +
-         #################################################################​ +
-         #################################################################​ +
-         #################################################################​ +
-         #################################################################​ +
-         #################################################################​ +
-         #################################################################​ +
-         #################################################################​ +
-         #################################################################​ +
-         #################################################################​ +
-         #################################################################​ +
-         #################################################################​ +
-         #################################################################​ +
-         #################################################################​ +
-         #################################################################​ +
-         #############################​ +
-done +
-Bytes transferred = 16121856 ​(f60000 hex+
-</​code>​ +
- +
-The firmware image is stored in RAM now and occupies **0xf60000** bytesWriting it from there to flash memory directly may fail when the target area has not been erased or if it is write-protected. Let's take care of that first. +
-==== Step 4Reset partition table to defaults ==== +
- +
-This gives you an overview of the partition table layout +
- +
-<​code>​ +
-ar7240> mtdparts +
-</​code>​ +
- +
-<​code>​ +
-device nor0 <​ath-nor0>,​ # parts = 5 +
- #name                        size            offset ​         mask_flags +
- 0u-boot ​                     0x00040000 ​     0x00000000 ​     0 +
- 1u-boot-env ​                 0x00010000 ​     0x00040000 ​     0 +
- 2jffs2                       ​0x00f60000 ​     0x00050000 ​     0 +
- 3cfg                         ​0x00040000 ​     0x00fb0000 ​     0 +
- 4EEPROM ​                     0x00010000 ​     0x00ff0000 ​     ​0 +
- +
-active partition: nor0,0 - (u-boot0x00040000 @ 0x00000000 +
- +
-defaults: +
-mtdids ​ : nor0=ath-nor0 +
-mtdpartsmtdparts=ath-nor0:​256k(u-boot),64k(u-boot-env),15744k(jffs2),​256k(cfg),​64k(EEPROM) +
-</code> +
- +
-Reset partition table to defaults with +
- +
-<​code>​ +
-ar7240> mtdparts default +
-</​code>​ +
- +
-Save environment changes: +
- +
-<​code>​ +
-ar7240> saveenv +
-</​code>​ +
- +
-<​code>​ +
-Saving Environment to Flash..+
-Un-Protected 1 sectors +
-Erasing Flash.... done +
-Erased ​sectors +
-Writing to Flash... write addr9f040000 +
-done +
-Protected 1 sectors +
-</code> +
- +
-==== Step 5: Unlock Flash-Banks ==== +
- +
-Now it is time to make sure that any write-protection is disabled: +
- +
-<​code>​ +
-ar7240> protect off all +
-</​code>​ +
-<​code>​ +
-Un-Protect Flash Bank # 1 +
-</code> +
- +
-==== Step 6Erase the corrupt jffs2 ==== +
- +
-Figuring out the correct address and length was a bit difficult, uncertain, not well documented and took us quite a while, because we didn't want to brick it more by writing stuff somewhere other than we intendedUnfortunately,​ there was no documentation about the process of how we derived at that conclusion, but in the end we finally decided to start at address **0x9f050000** for the length of the jffs2 image as indicated by the tftp transfer (**0xf60000**) and the mtd partition size (**0xf60000**) +
- +
-<​code>​ +
-ar7240> erase 0x9f050000 +0xf60000 +
-</​code>​ +
- +
-<​code>​ +
-...................................................................................................................................................................................................................................................... done +
-Erased 246 sectors +
-</​code>​ +
- +
-==== Step 7Flash original jffs2 ==== +
- +
-The cp command knows about flash memory areas and will automatically invoke the necessary flash programming algorithm when the target area is in flash memory. +
- +
-<​code>​ +
-ar7240> cp.b 0x83000000 0x9f050000 0xf60000 +
-</​code>​ +
- +
-<​code>​ +
-Copy to Flash... write addr9f050000 +
-done +
-</​code>​ +
- +
-==== Step 8Verify the jffs2 was flashed correctly ==== +
- +
-Since we still have the jffs2 image stored in RAM, starting at address **0x83000000**,​ we can now comfortably ​byte-compare it with the content starting at address **0x9f050000** ​(Flash) for the length of **0xf60000**Both have to match, otherwise something obviously went wrong during the flashing process. +
- +
-<​code>​ +
-ar7240> cmp.b 0x83000000 0x9f050000 0xf60000 +
-</code> +
-<​code>​ +
-Total of 16121856 bytes were the same +
-</​code>​ +
- +
-==== Step 9: Reset, Reinstall OpenWRT and enjoy your revival work ==== +
-<​code>​ +
-ar7240> reset +
-</​code>​ +
- +
-<​code>​ +
-Resetting... +
- +
-U-Boot unifi-v1.2.3.74-ge597862a (Mar 14 2013 - 18:30:11) +
- +
-BoardUbiquiti Networks AR9344 board (e507-27.2122.0030.0030) +
-DRAM 128 MB +
-Flash16 MB +
-Net  eth0 +
-Hit any key to stop autoboot 0 +
-Scanning JFFS2 FS. done. +
-## Booting image at 81000000 ​... +
-   Image Name  MIPS Ubiquiti Linux-2.6.32.33 +
-   ​Created     ​2014-06-06 ​ 23:26:31 UTC +
-   Image Type  MIPS Linux Kernel Image (lzma compressed) +
-   Data Size:    4450109 Bytes =  4.2 MB +
-   Load Address80002000 +
-   Entry Point ​80002000 +
-   ​Verifying Checksum at 0x81000040 ...OK +
-   ​Uncompressing Kernel Image ... OK +
- +
-Starting kernel ...+
 </​code>​ </​code>​