This post explains you the major difference between VMFS 3 and VMFS 5. VMFS 5 is available as part of vSphere 5. VMFS 5 is introduced with lot of performance enhancements. Newly installed ESXi 5 will be formatted with VMFS 5 version but if you have upgraded the ESX 4 or ESX 4.1 to ESXi 5, then datastore version will be VMFS 3 only. You will able to upgrade the VMFS 3 to VMFS 5 via vSphere client once ESXi upgrade is Complete. This posts tells you some major differences between VMFS 3 and VMFS 5
| Capability | VMFS 3 | VMFS 5 |
| Maximum single Extend size | 2 TB less 512 bytes | 64 TB |
| Partition Style | MBR (Master Boot Record) style | GPT (GUID Partition Table) |
| Available Block Size | 1 MB/2MB/4MB/8MB | only 1 MB |
| Maximum size of RDM in Virtual Compatibiltiy |
2 TB less 512 bytes | 2 TB less 512 bytes |
| Maximum size of RDM in Phsical Compatibiltiy |
2 TB less 512 bytes | 64 TB |
| Supported Hosts versions | ESX/ESX 3.X, 4.X & 5.x | Only ESXi 5 is supported |
| Spanned Volume size | 64 TB (32 extends with max size of extent is 2 TB) |
64 TB (32 extends with any size combination) |
| Upgrade path | VMFS 3 to VMFS 5 | Latest Version. NO upgarde available yet. |
| File Limit | 30,000 | 100,000 |
| Sub-Block size | 64 KB | 8 KB |