locked
hv_balloon: hot_add memory failed error is -12 RRS feed

  • Question

  • Oracle Linux 6.5

    Oracle Linux Server Unbreakable Enterprise Kernel (3.8.13-44.1.1.el6uek.x86_64)

    [root@node1 /]# cat /etc/udev/rules.d/100-balloon.rules
    SUBSYSTEM=="memory", ACTION=="add", ATTR{state}="online"

    dmesg:

    .../...

    Mem-Info:
    Node 0 DMA per-cpu:
    CPU    0: hi:    0, btch:   1 usd:   0
    Node 0 DMA32 per-cpu:
    CPU    0: hi:   90, btch:  15 usd:   0
    active_anon:5307 inactive_anon:6072 isolated_anon:0
     active_file:688 inactive_file:685 isolated_file:0
     unevictable:0 dirty:91 writeback:0 unstable:0
     free:2060 slab_reclaimable:2887 slab_unreclaimable:10478
     mapped:233 shmem:0 pagetables:571 bounce:0
     free_cma:0
    Node 0 DMA free:1184kB min:120kB low:148kB high:180kB active_anon:144kB inactive_anon:136kB active_file:8kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15648kB managed:15904kB mlocked:0kB dirty:0kB writeback:0kB mapped:4kB shmem:0kB slab_reclaimable:12kB slab_unreclaimable:28kB kernel_stack:16kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:40 all_unreclaimable? no
    lowmem_reserve[]: 0 236 236 236
    Node 0 DMA32 free:7056kB min:1904kB low:2380kB high:2856kB active_anon:21084kB inactive_anon:24152kB active_file:2744kB inactive_file:2740kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:241856kB managed:199012kB mlocked:0kB dirty:364kB writeback:0kB mapped:928kB shmem:0kB slab_reclaimable:11536kB slab_unreclaimable:41884kB kernel_stack:1384kB pagetables:2284kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
    lowmem_reserve[]: 0 0 0 0
    Node 0 DMA: 26*4kB (EM) 6*8kB (UEM) 3*16kB (UM) 3*32kB (UE) 4*64kB (UEM) 5*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1192kB
    Node 0 DMA32: 1187*4kB (UEM) 185*8kB (UEM) 50*16kB (EM) 1*32kB (M) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 7060kB
    3387 total pagecache pages
    2013 pages in swap cache
    Swap cache stats: add 1733535, delete 1731522, find 721341/899123
    Free swap  = 3956392kB
    Total swap = 4095996kB
    65504 pages RAM
    6172 pages reserved
    264147 pages shared
    53026 pages non-shared
    hv_balloon: hot_add memory failed error is -12
    hv_balloon: Memory hot add failed
    init_memory_mapping: [mem 0x18000000-0x1fffffff]
     [mem 0x18000000-0x1fffffff] page 2M
    kworker/0:1: page allocation failure: order:9, mode:0x80d0
    Pid: 5507, comm: kworker/0:1 Tainted: G        W    3.8.13-44.1.1.el6uek.x86_64 #2
    Call Trace:
     [<ffffffff81135513>] warn_alloc_failed+0xf3/0x160
     [<ffffffff81138310>] ? __alloc_pages_direct_compact+0x90/0x200
     [<ffffffff810c0423>] ? on_each_cpu_mask+0x53/0x70
     [<ffffffff81138926>] __alloc_pages_slowpath+0x4a6/0x7b0
     [<ffffffff81138f2b>] __alloc_pages_nodemask+0x2fb/0x320
     [<ffffffff81593e94>] vmemmap_alloc_block+0x63/0xd3
     [<ffffffff815940fa>] vmemmap_alloc_block_buf+0x30/0x3b
     [<ffffffff8159157a>] vmemmap_populate+0x12d/0x251
     [<ffffffff81593dc3>] sparse_mem_map_populate+0x2a/0x3c
     [<ffffffff81593bcc>] sparse_add_one_section+0x68/0x13b
     [<ffffffff815947bf>] __add_section+0x61/0xa7
     [<ffffffff81583e0a>] __add_pages+0x6a/0x90
     [<ffffffff8104b561>] arch_add_memory+0x71/0xf0
     [<ffffffff81583fba>] add_memory+0xba/0x1d0
     [<ffffffffa0235736>] hv_mem_hot_add+0xa6/0x140 [hv_balloon]
     [<ffffffffa0235859>] handle_pg_range+0x89/0x150 [hv_balloon]
     [<ffffffffa0235990>] process_hot_add+0x70/0x140 [hv_balloon]
     [<ffffffffa0235ba7>] hot_add_req+0x147/0x160 [hv_balloon]
     [<ffffffff8107bbc0>] process_one_work+0x180/0x420
     [<ffffffff8107dbfe>] worker_thread+0x12e/0x390
     [<ffffffff8107dad0>] ? manage_workers+0x180/0x180
     [<ffffffff81082f4e>] kthread+0xce/0xe0
     [<ffffffff81082e80>] ? kthread_freezable_should_stop+0x70/0x70
     [<ffffffff815a09ac>] ret_from_fork+0x7c/0xb0
     [<ffffffff81082e80>] ? kthread_freezable_should_stop+0x70/0x70
    Mem-Info:
    Node 0 DMA per-cpu:
    CPU    0: hi:    0, btch:   1 usd:   0
    Node 0 DMA32 per-cpu:
    CPU    0: hi:   90, btch:  15 usd:   0
    active_anon:6160 inactive_anon:6871 isolated_anon:32
     active_file:191 inactive_file:181 isolated_file:0
     unevictable:0 dirty:0 writeback:7 unstable:0
     free:1543 slab_reclaimable:2800 slab_unreclaimable:10397
     mapped:133 shmem:0 pagetables:571 bounce:0
     free_cma:0
    Node 0 DMA free:1100kB min:120kB low:148kB high:180kB active_anon:144kB inactive_anon:180kB active_file:8kB inactive_file:44kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15648kB managed:15904kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:8kB slab_unreclaimable:28kB kernel_stack:16kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
    lowmem_reserve[]: 0 236 236 236
    Node 0 DMA32 free:5072kB min:1904kB low:2380kB high:2856kB active_anon:24496kB inactive_anon:27304kB active_file:756kB inactive_file:680kB unevictable:0kB isolated(anon):128kB isolated(file):0kB present:241856kB managed:199012kB mlocked:0kB dirty:0kB writeback:32kB mapped:532kB shmem:0kB slab_reclaimable:11192kB slab_unreclaimable:41560kB kernel_stack:1384kB pagetables:2284kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
    lowmem_reserve[]: 0 0 0 0
    Node 0 DMA: 4*4kB (M) 4*8kB (UM) 4*16kB (UM) 3*32kB (UE) 4*64kB (UEM) 5*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1104kB
    Node 0 DMA32: 830*4kB (UEM) 111*8kB (UEM) 48*16kB (EM) 3*32kB (M) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 5072kB
    2794 total pagecache pages
    2434 pages in swap cache
    Swap cache stats: add 1737324, delete 1734890, find 723647/901845
    Free swap  = 3961464kB
    Total swap = 4095996kB
    65504 pages RAM
    6172 pages reserved
    263070 pages shared
    54538 pages non-shared
    hv_balloon: hot_add memory failed error is -12
    hv_balloon: Memory hot add failed
    init_memory_mapping: [mem 0x18000000-0x1fffffff]
     [mem 0x18000000-0x1fffffff] page 2M
    kworker/0:1: page allocation failure: order:9, mode:0x80d0
    Pid: 5507, comm: kworker/0:1 Tainted: G        W    3.8.13-44.1.1.el6uek.x86_64 #2
    Call Trace:
     [<ffffffff81135513>] warn_alloc_failed+0xf3/0x160
     [<ffffffff8113847a>] ? __alloc_pages_direct_compact+0x1fa/0x200
     [<ffffffff81138926>] __alloc_pages_slowpath+0x4a6/0x7b0
     [<ffffffff81138f2b>] __alloc_pages_nodemask+0x2fb/0x320
     [<ffffffff81593e94>] vmemmap_alloc_block+0x63/0xd3
     [<ffffffff815940fa>] vmemmap_alloc_block_buf+0x30/0x3b
     [<ffffffff8159157a>] vmemmap_populate+0x12d/0x251
     [<ffffffff81593dc3>] sparse_mem_map_populate+0x2a/0x3c
     [<ffffffff81593bcc>] sparse_add_one_section+0x68/0x13b
     [<ffffffff815947bf>] __add_section+0x61/0xa7
     [<ffffffff81583e0a>] __add_pages+0x6a/0x90
     [<ffffffff8104b561>] arch_add_memory+0x71/0xf0
     [<ffffffff81583fba>] add_memory+0xba/0x1d0
     [<ffffffffa0235736>] hv_mem_hot_add+0xa6/0x140 [hv_balloon]
     [<ffffffffa0235859>] handle_pg_range+0x89/0x150 [hv_balloon]
     [<ffffffffa0235990>] process_hot_add+0x70/0x140 [hv_balloon]
     [<ffffffffa0235ba7>] hot_add_req+0x147/0x160 [hv_balloon]
     [<ffffffff8107bbc0>] process_one_work+0x180/0x420
     [<ffffffff8107dbfe>] worker_thread+0x12e/0x390
     [<ffffffff8107dad0>] ? manage_workers+0x180/0x180
     [<ffffffff81082f4e>] kthread+0xce/0xe0
     [<ffffffff81082e80>] ? kthread_freezable_should_stop+0x70/0x70
     [<ffffffff815a09ac>] ret_from_fork+0x7c/0xb0
     [<ffffffff81082e80>] ? kthread_freezable_should_stop+0x70/0x70
    Mem-Info:
    Node 0 DMA per-cpu:
    CPU    0: hi:    0, btch:   1 usd:   0
    Node 0 DMA32 per-cpu:
    CPU    0: hi:   90, btch:  15 usd:   0
    active_anon:5503 inactive_anon:6074 isolated_anon:4
     active_file:464 inactive_file:464 isolated_file:0
     unevictable:0 dirty:38 writeback:0 unstable:0
     free:2355 slab_reclaimable:2822 slab_unreclaimable:10480
     mapped:246 shmem:0 pagetables:571 bounce:0
     free_cma:0
    Node 0 DMA free:1188kB min:120kB low:148kB high:180kB active_anon:80kB inactive_anon:200kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15648kB managed:15904kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:12kB slab_unreclaimable:28kB kernel_stack:16kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:52 all_unreclaimable? no
    lowmem_reserve[]: 0 236 236 236
    Node 0 DMA32 free:8232kB min:1904kB low:2380kB high:2856kB active_anon:21932kB inactive_anon:24096kB active_file:1856kB inactive_file:1856kB unevictable:0kB isolated(anon):16kB isolated(file):0kB present:241856kB managed:199012kB mlocked:0kB dirty:152kB writeback:0kB mapped:984kB shmem:0kB slab_reclaimable:11276kB slab_unreclaimable:41892kB kernel_stack:1384kB pagetables:2284kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
    lowmem_reserve[]: 0 0 0 0
    Node 0 DMA: 26*4kB (M) 4*8kB (UEM) 6*16kB (UEM) 4*32kB (UEM) 3*64kB (UM) 5*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1192kB
    Node 0 DMA32: 1306*4kB (EM) 258*8kB (UEM) 60*16kB (UEM) 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 8248kB
    3134 total pagecache pages
    2210 pages in swap cache
    Swap cache stats: add 1834737, delete 1832527, find 764692/953005
    Free swap  = 3956436kB
    Total swap = 4095996kB
    65504 pages RAM
    6172 pages reserved
    264363 pages shared
    52799 pages non-shared
    hv_balloon: hot_add memory failed error is -12
    hv_balloon: Memory hot add failed

    • Moved by BrianEhMVP Wednesday, October 15, 2014 10:30 PM
    Wednesday, October 15, 2014 9:47 PM

All replies

  • It seems after a Linux VM reboot hot-add is working, but... every time it reaches 500 MB it raises a kernel panic. Minimum memory is set to 128 MB, maximum 4096 MB, startup 256 MB. I don't know what information to collect about the kernel panic.

    In /var/log/messages before the kernel panic I see:

    Oct 16 01:31:53 node1 rpc.statd[1181]: Version 1.2.3 starting
    Oct 16 01:31:53 node1 sm-notify[1182]: Version 1.2.3 starting
    Oct 16 01:32:27 node1 kernel: hv_balloon: Received INFO_TYPE_MAX_PAGE_CNT
    Oct 16 01:32:27 node1 kernel: hv_balloon: Data Size is 8
    Oct 16 01:40:04 node1 kernel: init_memory_mapping: [mem 0x18000000-0x1fffffff]
    Oct 16 01:40:04 node1 kernel: Built 1 zonelists in Node order, mobility grouping on.  Total pages: 63617
    Oct 16 01:40:04 node1 kernel: Policy zone: DMA32
    Oct 16 01:40:04 node1 kernel: Built 1 zonelists in Node order, mobility grouping on.  Total pages: 96385
    Oct 16 01:40:04 node1 kernel: Policy zone: Normal
    Oct 16 01:40:22 node1 kernel: init_memory_mapping: [mem 0x20000000-0x27ffffff]
    Oct 16 01:40:22 node1 udevd-work[1343]: error opening ATTR{/sys/devices/system/memory/memory4/state} for writing: No such file or directory
    Oct 16 01:41:35 node1 kernel: nr_pdflush_threads exported in /proc is scheduled for removal
    Oct 16 01:41:35 node1 kernel: sysctl: The scan_unevictable_pages sysctl/node-interface has been disabled for lack of a legitimate use case.  If you have one, please send an email to linux-mm@kvack.org.

    Indeed, /sys/devices/system/memory/memory4 does not exists (only memory1 and memory2).

    Wednesday, October 15, 2014 11:00 PM
  • It seems hot-add does not work with UEK kernel, but only with RedHat compatible kernel. Oracle has not done it's homework. The problem with the RedHat kernel is that it is shipped with 2.6.32, while Oracle RDBMS 12cR1 requires at least 2.6.39..... and then the prerequisites rpm it installs the UEK which does not support hot-add....
    Friday, October 17, 2014 1:31 PM
  • Check the documentation here:  http://technet.microsoft.com/en-us/library/dn609828.aspx.  In the grid on the row for "Dynamic Memory -- Hot Add", you'll see that this is not supported for Oracle Linux with the UEK.   We are working with Oracle to get this added in a coming Quarterly Update for the UEKR3, but I can't promise yet exactly when that will happen.

    Michael Kelley, Lead Program Manager, Open Source Technology Center

    Friday, October 17, 2014 3:32 PM
    Moderator