SMART error detected on host

imok · November 30, 2018, 9:30pm

Hi
Received this message today:

The following warning/error was logged by the smartd daemon:
Device: /dev/sdc [SAT], Failed SMART usage Attribute: 190 Airflow_Temperature_Cel.

Run smartctl and I got this:

~# smartctl -a /dev/sdc

smartctl 6.6 2016-05-31 r4324 [x86_64-linux-4.4.59-1-pve] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Seagate Constellation.2 (SATA)
Device Model:     ST9250610NS
Serial Number:    9XE0CP3W
LU WWN Device Id: 5 000c50 063fa2ccd
Firmware Version: SN03
User Capacity:    250.059.350.016 bytes [250 GB]
Sector Size:      512 bytes logical/physical
Rotation Rate:    7200 rpm
Form Factor:      2.5 inches
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ATA8-ACS T13/1699-D revision 4
SATA Version is:  SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Fri Nov 30 16:26:11 2018 PET
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
See vendor-specific Attribute list for marginal Attributes.

General SMART Values:
Offline data collection status:  (0x82) Offline data collection activity
                    was completed without error.
                    Auto Offline Data Collection: Enabled.
Self-test execution status:      (   0) The previous self-test routine completed
                    without error or no self-test has ever 
                    been run.
Total time to complete Offline 
data collection:        (  650) seconds.
Offline data collection
capabilities:            (0x7b) SMART execute Offline immediate.
                    Auto Offline data collection on/off support.
                    Suspend Offline collection upon new
                    command.
                    Offline surface scan supported.
                    Self-test supported.
                    Conveyance Self-test supported.
                    Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                    power-saving mode.
                    Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                    General Purpose Logging supported.
Short self-test routine 
recommended polling time:    (   1) minutes.
Extended self-test routine
recommended polling time:    (  54) minutes.
Conveyance self-test routine
recommended polling time:    (   2) minutes.
SCT capabilities:          (0x10bd) SCT Status supported.
                    SCT Error Recovery Control supported.
                    SCT Feature Control supported.
                    SCT Data Table supported.

SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000f   081   063   044    Pre-fail  Always       -       143518405
  3 Spin_Up_Time            0x0003   097   096   000    Pre-fail  Always       -       0
  4 Start_Stop_Count        0x0032   100   100   020    Old_age   Always       -       24
  5 Reallocated_Sector_Ct   0x0033   100   100   036    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x000f   091   060   030    Pre-fail  Always       -       1349484709
  9 Power_On_Hours          0x0032   054   054   000    Old_age   Always       -       40458
 10 Spin_Retry_Count        0x0013   100   100   097    Pre-fail  Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   020    Old_age   Always       -       24
184 End-to-End_Error        0x0032   100   100   099    Old_age   Always       -       0
187 Reported_Uncorrect      0x0032   100   100   000    Old_age   Always       -       0
188 Command_Timeout         0x0032   100   100   000    Old_age   Always       -       0
189 High_Fly_Writes         0x003a   100   100   000    Old_age   Always       -       0
190 Airflow_Temperature_Cel 0x0022   045   045   045    Old_age   Always   FAILING_NOW 55 (Min/Max 33/55)
191 G-Sense_Error_Rate      0x0032   100   100   000    Old_age   Always       -       0
192 Power-Off_Retract_Count 0x0032   100   100   000    Old_age   Always       -       17
193 Load_Cycle_Count        0x0032   096   096   000    Old_age   Always       -       8518
194 Temperature_Celsius     0x0022   055   055   000    Old_age   Always       -       55 (0 17 0 0 0)
195 Hardware_ECC_Recovered  0x001a   117   099   000    Old_age   Always       -       143518405
197 Current_Pending_Sector  0x0012   100   100   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0010   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x003e   200   200   000    Old_age   Always       -       0

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed without error       00%     25473         -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

Should I be worried because Airflow_Temperature_Cel says FAILING NOW?

UPDATE: Received another message for a second disk with the same situation

Mason · November 30, 2018, 9:39pm

My guess is a fan in your case is dead/dying. You can install the lm-sensors package to get a better idea of what temps the rest of the sensors are at (CPU, each disk, etc.). That might help you get a better idea if your server is running hot.

If it is, then raise a ticket with your provider

imok · November 30, 2018, 9:49pm

I got this:

~# sensors
acpitz-virtual-0
Adapter: Virtual device
temp1:        +27.8°C  (crit = +105.0°C)
temp2:        +29.8°C  (crit = +105.0°C)

coretemp-isa-0000
Adapter: ISA adapter
Physical id 0:  +65.0°C  (high = +80.0°C, crit = +100.0°C)
Core 0:         +59.0°C  (high = +80.0°C, crit = +100.0°C)
Core 1:         +55.0°C  (high = +80.0°C, crit = +100.0°C)
Core 2:         +64.0°C  (high = +80.0°C, crit = +100.0°C)
Core 3:         +58.0°C  (high = +80.0°C, crit = +100.0°C)

So… same temperature as disks.

This is from Heymman. You have one? Yours should be next to mine with similar temperatures

Mason · November 30, 2018, 10:02pm

Hmm… CPU seems fine (about the same as my server).

But 55 degrees is pretty hot for a drive to be getting. Worth opening a ticket to let Francis know something might be up.

imok · December 1, 2018, 4:02pm

Received a reply about a cooling unit failing in the datacenter and some servers were automatically powered off, fortunately the Supermicro ones weren’t affected.

I see temperature is under 40 C now.