DLFileVersion inconsistency cause unique constrain violation on publishing

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: No Longer Reproducible
• Affects Version/s: 5.2.3
• Fix Version/s: 6.2.0 CE GA1
• Component/s: Core Infrastructure, DM, Fault Tolerance, Fault Tolerance > Clustering Framework, Performance, Staging, Staging > Export/Import, WCM
• Labels: None
• Environment: Oracle 11g, Weblogic 11g, RedHat 5 (x86_64)

We are experiencing a a data inconsistency in the database that cause any publication from stage
to live to fail with an unique constrain violation on the DLFileVerion table:
ORA-00001: unique constraint (SITIST.IX_6C5E6512) violated

Everytime an editor tried to publish to live was getting a white page with the error:
"Manage pages not available".

On the logs a generic ORMException, the cause was the ORA-00001 error above.

We identified the cause of the issue being the existance of DLFileVersion equals to the DLFileEntry
version.

Specifically executing these query we get results and we shouldn't had:
select * from dlfileentry dlentr left join dlfileversion dlvers on dlentr.folderId = dlvers.folderId and dlentr.name = dlvers.name where dlentr.version = dlvers.version;

We then operated from the web client on the document library portlet, directly on live, removing the
exceding versions of the three files having the issue and the pubblication was possible again.

Note: this step wasn't easy at all.. since there was an online cluster of 4 servers, we had to try
the operation several times before being actually able to publish without error, we theorized the previous
value was being kept in cache by other instances of the server.

There was another issue in the portal concerning another liferay issue about DLFileRank: #LPS-4639

It was causing the 3 home page flash to disappear very often.

The same three files was the one with the DLFileVersion issue described above.

We do not think this is a coincidence. We think that publishing the website while the DLFileRank
exception was happening either caused the addition of a wrong DLFileVersion either the incorrect
update of the DLFileEntry.

We can't give you a complete step-by-step to reproduce this issue because many random concurrent
event mixed togher caused the issue, naming some of them:

• DLFileRank issue from multiple users downloading the homepage flash video
• Publishing a big website with lot of files, images, webcontents
• possibly other website editor adding files or updating them.

We disabled the file ranking now and applied the patch described in the LPS-4639 currently on
testing because we believe this was the root cause of all.

We also applied a workaround to the DLFileVersion code that we think could prevent any other issue
of this type to ever happen again and we'd like to share it with the community.

We were unable to reproduce the issue and we'd like to have some feedback from a liferay developer
to validate our hyphotesis or give another feasible explanation on the flow of event the took us
into it.

The contributed patch does this modification to Liferay 5.2.3:

1) before creating a new DLFileVersion check if it already exist, if it
does then remove it an warn in the log. If it is a dirty cache issue
just warn.

2) Then check if on the filesystem the file exist and rename it before
writing the new one if it really exist warning again.

The patch is applicable within eclipse on the liferay source code choosing "ignore leading path name segments" = 1