Author Topic: Weird character generation  (Read 4546 times)

slavonec

  • Beginner
  • *
  • Posts: 4
Weird character generation
« on: April 28, 2015, 23:13:52 pm »
Hi there,

When I create product category or a product VM generates also 2 sets of additional characters on each side of the title. See attached image.
How can I fix that?

I'm using:
Joomla! 3.4.1
PHP Version 5.4.39
VirtueMart 3.0.8


Thank you in advance.

jenkinhill

  • UK Web Developer & Consultant
  • Global Moderator
  • Super Hero
  • *
  • Posts: 27466
  • Always on vacation
    • Jenkin Hill Internet
Re: Weird character generation
« Reply #1 on: April 28, 2015, 23:36:01 pm »
Did you copy and paste the title from Word or similar?
Kelvyn

Jenkin Hill Internet,
Lowestoft, Suffolk, UK

Unsolicited PMs/emails will be ignored.

Please mention your VirtueMart, Joomla and PHP versions when asking a question in this forum

Currently using VM 3.4.3.10057 on Joomla 3.9.10 PHP 7.0.33
Testing VM 3.5.0.10097 on Joomla 3.9.10

Jörgen

  • Global Moderator
  • Full Member
  • *
  • Posts: 2239
    • Kreativ Fotografi
  • VirtueMart Version: 3.4.x
Re: Weird character generation
« Reply #2 on: April 29, 2015, 08:04:43 am »
Hello slavonec

I suspect the same that Kelvyn does, You have got cut and paste with some extra features ;)
Looks like You have some carriage return (#13) and line feed (#10) inserted in Your text.
Just delete the strange characters and You will be fine.

regards

Jörgen @ Kreativ Fotografi
Joomla 3.9.8
Virtuemart 3.4.x
Olympiantheme Hera (customized)

lindapowers

  • Full Member
  • ***
  • Posts: 1335
  • If you're going through hell, keep going.
    • Venta de naranjas online y mandarinas
  • Skype Name: manu.gonzalez91
  • VirtueMart Version: Latest avi
Re: Weird character generation
« Reply #3 on: April 29, 2015, 11:41:07 am »
Dont know if this is related:



When storing the product name:

Oli d'oliva verge extra 250 ml

gets stored as:

Oli d' oliva verge extra 250 ml

The apostrophe is not welcomed. Strangely the product name appears correctly written everywhere except in the dropdown for the childs.


vm 3.08 j 3.4.1

slavonec

  • Beginner
  • *
  • Posts: 4
Re: Weird character generation
« Reply #4 on: April 29, 2015, 17:57:18 pm »
Well, in the beginning I thought that exactly what happened (copy and paste), but then I TYPED THE WORDS - same outcome!

I then, deleted all I created, flushed the database, reinstall VM fresh - same problem. My database charset us UTF-8, but just to stay safe I forced it also with my .htaccess - the problem persist.

So, put aside COPY & PASTE lets rethink what could have happen. I do not have idea.

Could it be that JCE editor is messing with VM?! Or something else.

Thank you

GJC Web Design

  • 3rd party VirtueMart Developer
  • Super Hero
  • *
  • Posts: 8891
  • Virtuemart, Joomla & php developer
    • GJC Web Design
  • VirtueMart Version: 3.4.2
Re: Weird character generation
« Reply #5 on: April 29, 2015, 21:33:27 pm »
have u tried a different or no editor?
GJC Web Design
VirtueMart and Joomla Developers - php developers http://www.gjcwebdesign.com
VM3 AusPost Shipping Plugin - e-go Shipping Plugin - VM3 Postcode Shipping Plugin - Radius Shipping Plugin - VM3 NZ Post Shipping Plugin - AusPost Estimator
Samport Payment Plugin - EcomMerchant Payment Plugin - ccBill payment Plugin
VM2 Product Lock Extension - VM2 Preconfig Adresses Extension - TaxCloud USA Taxes Plugin - Virtuemart  Product Review Component
http://extensions.joomla.org/profile/profile/details/67210
Contact for any VirtueMart or Joomla development & customisation

Milbo

  • Virtuemart Projectleader
  • Administrator
  • Super Hero
  • *
  • Posts: 9853
  • VM3.2 Cached and Optimized
    • VM3 Extensions
  • VirtueMart Version: VirtueMart 3 on joomla 3
Re: Weird character generation
« Reply #6 on: April 29, 2015, 22:21:44 pm »
Lindapowers, it is correctly stored that way, the question is now, why it does not show correcty in your dropdown. I assume you use the multivariant? and you do not work as superadmin? There is an ACL which allows you to use raw as input. Just enable it for your admins
I should fix your bug, please support the VirtueMart project and become a member
______________________________________
Extensions approved by the core team: http://extensions.virtuemart.net/

slavonec

  • Beginner
  • *
  • Posts: 4
Re: Weird character generation
« Reply #7 on: April 29, 2015, 22:57:49 pm »
Solved!

JCE Full Page Editing is inserting [prefix="og: http://ogp.me/ns#"]

More here: https://www.joomlacontenteditor.net/support/forum/jce-inserts-unneeded-html-tags

Thank you for the prompt action guys.


Thank you

lindapowers

  • Full Member
  • ***
  • Posts: 1335
  • If you're going through hell, keep going.
    • Venta de naranjas online y mandarinas
  • Skype Name: manu.gonzalez91
  • VirtueMart Version: Latest avi
Re: Weird character generation
« Reply #8 on: April 30, 2015, 12:06:48 pm »
Lindapowers, it is correctly stored that way, the question is now, why it does not show correcty in your dropdown. I assume you use the multivariant? and you do not work as superadmin? There is an ACL which allows you to use raw as input. Just enable it for your admins


Hello

I work as superadmin. In this case we are using generic child variants. I checked the ACL and the setting was allowed for admins also superadmins.

Regards

Jörgen

  • Global Moderator
  • Full Member
  • *
  • Posts: 2239
    • Kreativ Fotografi
  • VirtueMart Version: 3.4.x
Re: Weird character generation
« Reply #9 on: April 30, 2015, 12:58:14 pm »
@Milbo

This is a bug fix regarding the first issue, not the linda powers issue, but it could depend on the same code missing. If You have a better solution, please say.  I suggest this should be added to the next update. The text is not rendered properly when shown in the category view. (Joomla! 3.4.1, VM 3.0.6.4)

@slavonec

I see this in my backend to. But only with newly saved text. the JCE-editor seems to save the text html safe. This will not render good in the VM back end. This will make the text independent of how the editor stores the text :)

I have made an override for the category view:

administrator/components/com_virtuemart/views/category/tmpl/default.php

Copy the file to:
administrator/templates/your-adm-template/html/com_virtuemart/category/default.php  (in my case ISIS)

Find line 131
Code: [Select]

echo shopFunctionsF::limitStringByWord(JFilterOutput::cleanText($cat->category_description),200); ?>

And replace with
Code: [Select]

echo shopFunctionsF::limitStringByWord(htmlspecialchars_decode(JFilterOutput::cleanText($cat->category_description)),200); ?>

There will probably be lots of other places where the htmlspecialchars_decode() should be added.

regards

Jörgen @ Kreativ Fotografi



Joomla 3.9.8
Virtuemart 3.4.x
Olympiantheme Hera (customized)

Jörgen

  • Global Moderator
  • Full Member
  • *
  • Posts: 2239
    • Kreativ Fotografi
  • VirtueMart Version: 3.4.x
Re: Weird character generation
« Reply #10 on: May 01, 2015, 19:17:06 pm »
I have found more places for the htmlspecialchars_decode(). In the view orderstatus:

change line 97:
Code: [Select]
<?php echo vmText::_($row->order_status_description); ?>to
Code: [Select]
<?php echo htmlspecialchars_decode(vmText::_($row->order_status_description)); ?>
Make an override as suggested in my previous post until the VM team has corrected this.
As soon as I find more I will append them here.

regards

Jörgen @ Kreativ Fotografi


Joomla 3.9.8
Virtuemart 3.4.x
Olympiantheme Hera (customized)

Milbo

  • Virtuemart Projectleader
  • Administrator
  • Super Hero
  • *
  • Posts: 9853
  • VM3.2 Cached and Optimized
    • VM3 Extensions
  • VirtueMart Version: VirtueMart 3 on joomla 3
Re: Weird character generation
« Reply #11 on: May 01, 2015, 22:15:32 pm »
No, this is wrong technic. Vm stores encrypted, when you do an encode => you open your vm for persistent XSS attacks!

The topic is very, very complex. It starts with that you cannot use your browser to understand what is going on, because your browser IS usually directly translating html entities. What is the trick with it?
Lets explain it that way. Assume a + is a dangerous sign, then VM stores the + as a cross, looks like the +, but has not the effect of it.

Btw, I just tested it. In my case the ' and ö remains. Maybe the ACL is not correctly set in your case (migrated). The idea is that admins are allowed to store raw or html filtered, but others not. Then it is directly stored encoded. Therefore the output must not be encoded or decoded. Except the js, but that is another story. JS has the nasty effect to interprete a "cross" as "plus"


I should fix your bug, please support the VirtueMart project and become a member
______________________________________
Extensions approved by the core team: http://extensions.virtuemart.net/

Jörgen

  • Global Moderator
  • Full Member
  • *
  • Posts: 2239
    • Kreativ Fotografi
  • VirtueMart Version: 3.4.x
Re: Weird character generation
« Reply #12 on: May 02, 2015, 13:19:55 pm »
@Milbo

Thanks for Yor reply.

The places where I applied the fix are in the back end, but OK I can see Your concern with XSS.

I have messed around and tried with different editors and now the characters seem to save as UTF-8 and nothing else. This is strange and I have not been able to replicate the earlier behaviour. The issue with translated å,ä,ö has somehow magically vanished.

The remaining problem, for me anyway, seems to be the JFilterOutput::cleanText() function that gives me the &#13;(CR), &#10;(LF), &#9;.
I don´t seem to get rid of them. I tried tinyMCE and JCE editor. It looks like the text has a lot of CR,LF and using the JCE editor also gets tabular characters stored. They get translated by the JFilterOutput::cleanText() which is calling htmlspecialchars() that comes up with the decoded text.

Since the JFilterOutput::cleanText() isn´t beeing called in the front end there is no issue in the frontend, only in the back end. Front end looks fine.

Stripping out the &#13;, &#10; and &#9; in the back end works of course and this would absolutely not translate into any XSS vulnerability.

If I insert category descriptions that has in the Virtuemart live demo (3.0.8.0) I get &#13;&#10; for each linefeed so if there is some problem with the editor. The editor used in the latest Virtuemart demo certainly has the same issue.

Grateful for any suggestions.

regards

Jörgen @ Kreativ Fotografi
Joomla 3.9.8
Virtuemart 3.4.x
Olympiantheme Hera (customized)

Jörgen

  • Global Moderator
  • Full Member
  • *
  • Posts: 2239
    • Kreativ Fotografi
  • VirtueMart Version: 3.4.x
Re: Weird character generation
« Reply #13 on: May 03, 2015, 11:11:18 am »
Hello

This is what I have done. Hopefully it will help anyone who gets the &#13;, &#10; and &#9; characters stored and don´t want to see these in the backend. This should not give any XSS issues. I simply remove the characters except #13 which I replace with a space for clarity.

For line 97 in the admin view orderstatus
find
Code: [Select]
<?php echo vmText::_($row->order_status_description); ?>
replace with
Code: [Select]
<?php echo str_replace(["&amp;#13;","&amp;#10;","&amp;#9;"],[" ","",""], vmText::_($row->order_status_description)); 
For line 131 in the admin view category
find
Code: [Select]
echo shopFunctionsF::limitStringByWord(JFilterOutput::cleanText($cat->category_description),200);
replace with
Code: [Select]
echo shopFunctionsF::limitStringByWord(str_replace(["&amp;#13;","&amp;#10;","&amp;#9;"],[" ","",""],JFilterOutput::cleanText($cat->category_description)),200);
regards

Jörgen @ Kreativ Fotografi
Joomla 3.9.8
Virtuemart 3.4.x
Olympiantheme Hera (customized)

toocool

  • Beginner
  • *
  • Posts: 33
Re: Weird character generation
« Reply #14 on: May 03, 2015, 13:27:37 pm »
I have the same problem!

Only in VM. (3.0.8.) In admin weird, in the page source code weird. Only in the browser good.