As Xeon Phi is a quite new piece of hardware there are very few information in the Internet regarding its usage, installation and compatible hardware (especially motherboards). We would like to fill that gap and write about our experience of assembling custom server with Xeon Phi installed.
Technically, Xeon Phi looks very much like high-end 2-slot PCIe graphics card, except of the fact that it doesn't have DVI output on its back. We have 5110P model with passive cooling, hence there are models with active cooling. Without additional cooling there might be some temperature issues, so make sure that you have a case with additional cooler. Our server case has additional cooling installed and is placed in server room with powerful air conditioning, so no problems for us there.
Not every single motherboard can support this kind of accelerator. The trick is that it has to support 64-bit PCIe addressing and you can find this feature only on modern high-end motherboards. As Intel doesn't have official list of supported motherboards it could be quite tricky to choose motherboard for your Xeon Phi server, in case you buy it separately and assemble by yourself. We managed to run the accelerator with ASUS P9X79 WS motherboard (thanks ASUS for clear information that it supports Xeon Phi). Our motherboard had outdated version of BIOS, so we had to flush the latest version and change settings there in order to enable Xeon Phi support. If you visited a link above you might have noticed a statement that it supports only Xeon Phi 3100 series with active cooling, but in fact it works perfectly with 5110P, just don't forget about additional cooling in your case.
At peak loads Xeon Phi 5110P consumes 225W of power. 3100 series devices use even more, up to 300W, so make sure that your power-supply unit is able to feed it with that amount of energy.
Officially Xeon Phi works only with RHEL and SLES, but in our case it works perfectly with CentOS (open source clone of RHEL).
We are going to write more posts about Xeon Phi very soon. Stay tuned!
What processor did you use with ASUS P9X79 WS?
ReplyDelete"...and change settings there in order to enable Xeon Phi support"
what settings?
Core i7. Don't remember exactly, but I think it was Sandy Bridge once.
DeleteThere is a setting in BIOS with name like "Xeon Phi Support - Enabled/Disabled".
This comment has been removed by the author.
DeleteThis comment has been removed by the author.
DeleteThanks for the useful posting, Victor! After having tried our 5110P in several desktop'ish motherboards, unsuccessfully, we're now thinking of purchasing a new machine to run it in. It appears that motherboards supporting Xeon Phi generally use Intel's C600 or X79 chipset (I guess this is almost(?) the same chipset, marketed to different segments under these names), but there may also be issues with BIOS and with other peripherals. We have a few questions:
ReplyDeleteHow are you cooling your 5110P? What temperatures are you getting (per micinfo)? Here's our improvised cooling setup for ours, which we didn't get a chance to test yet (because of the motherboard compatibility issues): http://openwall.info/wiki/internal/xeon_phi#An-attempt-at-cooling-it
What case did you put the ASUS P9X79 WS motherboard in, given that it's SSI CEB form factor rather than ATX (although it's very similar)?
There's the ASUS ESC700 G2 platform from ASUS, which uses this motherboard, but we're unhappy with its bundled 500W PSU (not letting us install more accelerator cards in the future - whether MICs or GPUs - without upgrading the PSU) and its mediocre cooling. On the other hand, it appears to be the cheapest suitable platform out there (although we do have to invent our own cooling for the MIC card, unless we had a 3120A).
Some other suitable platforms (according to our "research" so far), for your blog readers:
SuperMicro:
http://www.supermicro.com/products/nfo/Xeon_Phi.cfm
http://www.supermicro.com/products/nfo/GPU_MIC.cfm
ASUS:
http://www.asus.com/Commercial_Servers_Workstations/ESC4000_G2/
Intel:
Xeon E5-2400 series: P4000SC, R2000BB
Xeon E5-2600 series: R1000JP, R2000GZ, P4000CO, P4000CR, P4000IP