VirtueMart Forum

VirtueMart 2 + 3 + 4 => Virtuemart Development and bug reports => Topic started by: slavonec on April 28, 2015, 23:13:52 PM

Title: Weird character generation
Post by: slavonec on April 28, 2015, 23:13:52 PM
Hi there,

When I create product category or a product VM generates also 2 sets of additional characters on each side of the title. See attached image.
How can I fix that?

I'm using:
Joomla! 3.4.1
PHP Version 5.4.39
VirtueMart 3.0.8


Thank you in advance.
Title: Re: Weird character generation
Post by: jenkinhill on April 28, 2015, 23:36:01 PM
Did you copy and paste the title from Word or similar?
Title: Re: Weird character generation
Post by: Jörgen on April 29, 2015, 08:04:43 AM
Hello slavonec

I suspect the same that Kelvyn does, You have got cut and paste with some extra features ;)
Looks like You have some carriage return (#13) and line feed (#10) inserted in Your text.
Just delete the strange characters and You will be fine.

regards

Jörgen @ Kreativ Fotografi
Title: Re: Weird character generation
Post by: lindapowers on April 29, 2015, 11:41:07 AM
Dont know if this is related:

(http://s13.postimg.org/5lqb16ed3/oliva2222.jpg)

When storing the product name:

Oli d'oliva verge extra 250 ml

gets stored as:

Oli d' oliva verge extra 250 ml

The apostrophe is not welcomed. Strangely the product name appears correctly written everywhere except in the dropdown for the childs.


vm 3.08 j 3.4.1
Title: Re: Weird character generation
Post by: slavonec on April 29, 2015, 17:57:18 PM
Well, in the beginning I thought that exactly what happened (copy and paste), but then I TYPED THE WORDS - same outcome!

I then, deleted all I created, flushed the database, reinstall VM fresh - same problem. My database charset us UTF-8, but just to stay safe I forced it also with my .htaccess - the problem persist.

So, put aside COPY & PASTE lets rethink what could have happen. I do not have idea.

Could it be that JCE editor is messing with VM?! Or something else.

Thank you
Title: Re: Weird character generation
Post by: GJC Web Design on April 29, 2015, 21:33:27 PM
have u tried a different or no editor?
Title: Re: Weird character generation
Post by: Milbo on April 29, 2015, 22:21:44 PM
Lindapowers, it is correctly stored that way, the question is now, why it does not show correcty in your dropdown. I assume you use the multivariant? and you do not work as superadmin? There is an ACL which allows you to use raw as input. Just enable it for your admins
Title: Re: Weird character generation
Post by: slavonec on April 29, 2015, 22:57:49 PM
Solved!

JCE Full Page Editing is inserting [prefix="og: http://ogp.me/ns#"]

More here: https://www.joomlacontenteditor.net/support/forum/jce-inserts-unneeded-html-tags (https://www.joomlacontenteditor.net/support/forum/jce-inserts-unneeded-html-tags)

Thank you for the prompt action guys.


Thank you
Title: Re: Weird character generation
Post by: lindapowers on April 30, 2015, 12:06:48 PM
Quote from: Milbo on April 29, 2015, 22:21:44 PM
Lindapowers, it is correctly stored that way, the question is now, why it does not show correcty in your dropdown. I assume you use the multivariant? and you do not work as superadmin? There is an ACL which allows you to use raw as input. Just enable it for your admins


Hello

I work as superadmin. In this case we are using generic child variants. I checked the ACL and the setting was allowed for admins also superadmins.

Regards
Title: Re: Weird character generation
Post by: Jörgen on April 30, 2015, 12:58:14 PM
@Milbo

This is a bug fix regarding the first issue, not the linda powers issue, but it could depend on the same code missing. If You have a better solution, please say.  I suggest this should be added to the next update. The text is not rendered properly when shown in the category view. (Joomla! 3.4.1, VM 3.0.6.4)

@slavonec

I see this in my backend to. But only with newly saved text. the JCE-editor seems to save the text html safe. This will not render good in the VM back end. This will make the text independent of how the editor stores the text :)

I have made an override for the category view:

administrator/components/com_virtuemart/views/category/tmpl/default.php

Copy the file to:
administrator/templates/your-adm-template/html/com_virtuemart/category/default.php  (in my case ISIS)

Find line 131

echo shopFunctionsF::limitStringByWord(JFilterOutput::cleanText($cat->category_description),200); ?>


And replace with

echo shopFunctionsF::limitStringByWord(htmlspecialchars_decode(JFilterOutput::cleanText($cat->category_description)),200); ?>


There will probably be lots of other places where the htmlspecialchars_decode() should be added.

regards

Jörgen @ Kreativ Fotografi



Title: Re: Weird character generation
Post by: Jörgen on May 01, 2015, 19:17:06 PM
I have found more places for the htmlspecialchars_decode(). In the view orderstatus:

change line 97:
<?php echo vmText::_($row->order_status_description); ?>
to
<?php echo htmlspecialchars_decode(vmText::_($row->order_status_description)); ?>

Make an override as suggested in my previous post until the VM team has corrected this.
As soon as I find more I will append them here.

regards

Jörgen @ Kreativ Fotografi


Title: Re: Weird character generation
Post by: Milbo on May 01, 2015, 22:15:32 PM
No, this is wrong technic. Vm stores encrypted, when you do an encode => you open your vm for persistent XSS attacks!

The topic is very, very complex. It starts with that you cannot use your browser to understand what is going on, because your browser IS usually directly translating html entities. What is the trick with it?
Lets explain it that way. Assume a + is a dangerous sign, then VM stores the + as a cross, looks like the +, but has not the effect of it.

Btw, I just tested it. In my case the ' and ö remains. Maybe the ACL is not correctly set in your case (migrated). The idea is that admins are allowed to store raw or html filtered, but others not. Then it is directly stored encoded. Therefore the output must not be encoded or decoded. Except the js, but that is another story. JS has the nasty effect to interprete a "cross" as "plus"


Title: Re: Weird character generation
Post by: Jörgen on May 02, 2015, 13:19:55 PM
@Milbo

Thanks for Yor reply.

The places where I applied the fix are in the back end, but OK I can see Your concern with XSS.

I have messed around and tried with different editors and now the characters seem to save as UTF-8 and nothing else. This is strange and I have not been able to replicate the earlier behaviour. The issue with translated å,ä,ö has somehow magically vanished.

The remaining problem, for me anyway, seems to be the JFilterOutput::cleanText() function that gives me the &#13;(CR), &#10;(LF), &#9;.
I don´t seem to get rid of them. I tried tinyMCE and JCE editor. It looks like the text has a lot of CR,LF and using the JCE editor also gets tabular characters stored. They get translated by the JFilterOutput::cleanText() which is calling htmlspecialchars() that comes up with the decoded text.

Since the JFilterOutput::cleanText() isn´t beeing called in the front end there is no issue in the frontend, only in the back end. Front end looks fine.

Stripping out the &#13;, &#10; and &#9; in the back end works of course and this would absolutely not translate into any XSS vulnerability.

If I insert category descriptions that has in the Virtuemart live demo (3.0.8.0) I get &#13;&#10; for each linefeed so if there is some problem with the editor. The editor used in the latest Virtuemart demo certainly has the same issue.

Grateful for any suggestions.

regards

Jörgen @ Kreativ Fotografi
Title: Re: Weird character generation
Post by: Jörgen on May 03, 2015, 11:11:18 AM
Hello

This is what I have done. Hopefully it will help anyone who gets the &#13;, &#10; and &#9; characters stored and don´t want to see these in the backend. This should not give any XSS issues. I simply remove the characters except #13 which I replace with a space for clarity.

For line 97 in the admin view orderstatus
find
<?php echo vmText::_($row->order_status_description); ?>

replace with
<?php echo str_replace(["&amp;#13;","&amp;#10;","&amp;#9;"],[" ","",""], vmText::_($row->order_status_description)); 

For line 131 in the admin view category
find
echo shopFunctionsF::limitStringByWord(JFilterOutput::cleanText($cat->category_description),200);

replace with
echo shopFunctionsF::limitStringByWord(str_replace(["&amp;#13;","&amp;#10;","&amp;#9;"],[" ","",""],JFilterOutput::cleanText($cat->category_description)),200);

regards

Jörgen @ Kreativ Fotografi
Title: Re: Weird character generation
Post by: toocool on May 03, 2015, 13:27:37 PM
I have the same problem!

Only in VM. (3.0.8.) In admin weird, in the page source code weird. Only in the browser good.
Title: Re: Weird character generation
Post by: toocool on May 03, 2015, 14:34:26 PM
I think I got the problem. In SystemInformation the database alignment is latin1_swedish_ci.
Title: Re: Weird character generation
Post by: lindapowers on May 03, 2015, 17:38:46 PM
Quote from: toocool on May 03, 2015, 14:34:26 PM
I think I got the problem. In SystemInformation the database alignment is latin1_swedish_ci.

I have that also.
Title: Re: Weird character generation
Post by: GJC Web Design on May 03, 2015, 19:35:58 PM
I always use utf8_general_ci.. 
Title: Re: Weird character generation
Post by: jenkinhill on May 03, 2015, 19:49:48 PM
Quote from: GJC Web Design on May 03, 2015, 19:35:58 PM
I always use utf8_general_ci.. 

Moi aussi.
Title: Re: Weird character generation
Post by: toocool on May 03, 2015, 20:02:33 PM
I modified to utf8 general ci but still bad. :/
Title: Re: Weird character generation
Post by: toocool on May 03, 2015, 21:19:02 PM
I looked another joomla content source codes. It's normal, so It is VM problem!
Title: Re: Weird character generation
Post by: Milbo on May 04, 2015, 19:19:30 PM
It works for others. And no one of you answered about the ACL setting. I think all the changes done by jören are unecessary.

btw the colllation of your table has nothing todo with it (it only affects sorting!)
Title: Re: Weird character generation
Post by: lindapowers on May 04, 2015, 19:33:45 PM
Quote from: Milbo on May 04, 2015, 19:19:30 PM
It works for others. And no one of you answered about the ACL setting. I think all the changes done by jören are unecessary.

btw the colllation of your table has nothing todo with it (it only affects sorting!)

I tested the ACL setting but im superadmin and all those were allowed already.
Title: Re: Weird character generation
Post by: Jörgen on May 05, 2015, 07:50:46 AM
@Milbo

I did a check in the official VM3 demo, the "problem" with visible #10 and #13 is there, so there could be something wrong with the settings for the demo user.
I understand that the demo user doesn´t have unlimited access. So maybe there is a clue here. If we are supposed to see &gt;, &#10; and &#13; in the backend, please tell us so. Don´t tell us that no one else has the problem, because the problem is visible in the demo. It is not a biggie, but still very annoying.

Quote from my earlier post:
QuoteIf I insert category descriptions that has in the Virtuemart live demo (3.0.8.0) I get &#13;&#10; for each linefeed so if there is some problem with the editor. The editor used in the latest Virtuemart demo certainly has the same issue.

Sorry that I haven´t answered about the ACL. I am logged in as superuser and can only see unlimited rights on all settings for the username.

With due respect, I will use my overrides until I find out the reason why this is happening .

regards

Jörgen @ Kreativ Fotografi


Title: Re: Weird character generation
Post by: Milbo on May 05, 2015, 10:34:26 AM
Exactly,
When you change as demo user something in the BE, then anything is stored encoded. The whole reason for this is to give untrusted users the possibility to create/edit products. But actually it should not be seen, there is somewhere a doubled encode.
Title: Re: Weird character generation
Post by: Jörgen on May 05, 2015, 10:45:22 AM
Thanks Milbo

Now we are going somewhere. I keep my overrides for now and move on with other issues. ;)

regards

Jörgen @ Kreativ Fotografi