header
header Register : : Login header
header
divider
menuleft
menuright
submenu
left

[August 25th, 2008] Check the home page regarding PowerShell related news from a brand new sponsor: Idera

Subject: Problem with where and -or
Prev Next
You are not authorized to post a reply.

Author Messages
frejenUser is Offline
New Member
New Member
Posts:8

07/22/2008 7:13 AM  

Hi,

Iam using the get-qaduser cmdlet in order to retrieve ad users. Then i want to pipe the returned user to where in order to look for some info in the Description field. I am trying to use the following line but it does not work as intended:

> get-qaduser -some things | where {($_.Description -inotmatch "SA:") -or ($_.Description -inotmatch "SC:")}

The problem is that users that have SA: or SC: in their descriptions will still appear in the final output. But if i do like this:

> get-qaduser -some things | where {($_.Description -inotmatch "SA:")} | where {($_.Description -inotmatch "SC:")}

Then it seem to do the job right. But why can i not use -or? I would prefer that over piping to where two times.

bsonposhUser is Offline
CLI Addict
CLI Addict
Posts:363

07/22/2008 7:31 AM  
your logic is flawed.

if you say if x not match y or not match x then it will be true cause it does it doesnt match y

You need -and here.
where(y -notmatch x -and y -notmatch y)

or, because -match is a regex

get-qaduser -some things | where {$_.Description -inotmatch "SA:|SC:"}




Brandon Shell
----------------
Microsoft Powershell MVP
https://mvp.support.microsoft.com/profile/Brandon
glnsizeUser is Offline
Shell Enthusiast
Shell Enthusiast
Posts:60

07/22/2008 11:29 AM  

I would suggest letting AD filter the data, try an LDAP filter like the one below.  PowerShell is great but nothing is faster than AD...

Get-QADUser -searchRoot 'fabricam.com/users' -SizeLimit 0 -LdapFilter `
 '(&(objectCategory=person)(objectClass=user)(!description=*SA:*)(!description=*SC:*))'

~Glenn

frejenUser is Offline
New Member
New Member
Posts:8

07/23/2008 3:31 AM  
bsonposh i guessed my logic could be flawed. But if i use -and will i not get further problems. If i use -and both matches has to fail in order to be true?

glnsize -LdapFilter switch seems interesting, think i can use it in quite many scripts ;-)

Thank you both
bsonposhUser is Offline
CLI Addict
CLI Addict
Posts:363

07/23/2008 5:28 AM  
Perhaps I misunderstand your end goal. are you saying you only want descriptions that match SA: or SC:? Or do you want only ones that do NOT match. I just want to make sure you understand the operators.

Either way... if your happy with the ldap filter that is good as well. It is really inefficient, but there is really no efficient way to do what you want.

Brandon Shell
----------------
Microsoft Powershell MVP
https://mvp.support.microsoft.com/profile/Brandon
glnsizeUser is Offline
Shell Enthusiast
Shell Enthusiast
Posts:60

07/23/2008 9:51 AM  

I was under the impression that LDAP filters were efficient...

bsonposhUser is Offline
CLI Addict
CLI Addict
Posts:363

07/23/2008 9:56 AM  
Generally speaking the are head and shoulders above client side filtering.

Although In this case, You are using two wild card searches. Depending on the size of the directory, that is alot of work for the DC.

To be honest, in this case, there is no real efficient way to do this.

Brandon Shell
----------------
Microsoft Powershell MVP
https://mvp.support.microsoft.com/profile/Brandon
halr9000User is Offline
CLI Addict
CLI Addict
Posts:245


07/23/2008 9:56 AM  
They are much more efficient than doing the filtering outside of LDAP (with where-object). But if you look at _this_ filter and apply a little bit of database theory, you can understand why Brandon said that. Freeform text fields are rarely indexed in a database, and searches like this one are expensive enough CPU-wise already. It's a matter of this: "find all user objects", versus this: "look at every single record, then toss out this and this".

does that help?

Community Director, PowerShellCommunity.org
Co-host, PowerScripting Podcast (http://powerscripting.net)
Author, TechProsaic (http://halr9000.com)
bsonposhUser is Offline
CLI Addict
CLI Addict
Posts:363

07/23/2008 10:07 AM  
btw.. you can use my Test-LDAPFilter.ps1 to see the performance from the DC

Here are the output of this script using this filter. Notice I had to touch every user.

Statistics
=================================
Elapsed Time: 70906.25 (ms)
Returned 202037 entries of 202037 visited - (100.00 %)

Used Filter:
-  ( &  (objectCategory=CN=Person,CN=Schema,CN=Configuration,,DC=bleh,DC=com)  (objectClass=user) ( !  (description=*SA
:*) ) ( !  (description=*SC:*) ) )

Used Indices:
- idx_objectCategory:70997:N;

Brandon Shell
----------------
Microsoft Powershell MVP
https://mvp.support.microsoft.com/profile/Brandon
bsonposhUser is Offline
CLI Addict
CLI Addict
Posts:363

07/23/2008 10:10 AM  
After further discussion (with Dean Wells)

We concluded that the LDAP Filter would be more efficient if the size of the return set was large and you had fairly decent DCs.

In such case you had a smaller return set (I would speculated >5k) then client side processing would probably win.

Brandon Shell
----------------
Microsoft Powershell MVP
https://mvp.support.microsoft.com/profile/Brandon
bsonposhUser is Offline
CLI Addict
CLI Addict
Posts:363

07/23/2008 10:12 AM  
ironically... this really has nothing to do with raw CPU usage, but more a factor of Network traffic.

With the filter, each user in the Return Set you would need to send the client the User object.
Without... you would need to send all the users.

p.s. Here is my blog entry about Test-LDAPFilter
http://bsonposh.com/archives/331

Brandon Shell
----------------
Microsoft Powershell MVP
https://mvp.support.microsoft.com/profile/Brandon
glnsizeUser is Offline
Shell Enthusiast
Shell Enthusiast
Posts:60

07/23/2008 6:09 PM  

@BSonPosh

cool, I haven't lost my mind or fallen out of touch :) ...  When you said efficent, I thought you had some crazy ADSI back door .net hook(wouldn't be the first time).  I do agree that wildcard searches are a last resort, but the speed of ldap filters just blow my mind. I tried to use where-object to filter efficently. After two days I gave up, and added an ldap filter, execution time went from 8 sec to >1.  Don't get me wrong there is nothing powershell can't do, but simmilar to a SQL query... How you ask for the data is more important then what you do with it.

@hal

Point taken, but I would still argue that the sooner you get data out of the pipeline the faster it will perform.  Regaurdless of how great ?{} is... there is a performace hit when asking it proccess 20,000 object vice 2,000.  I'm sure you already know that... now small to medium data sets Where/Select are the way to go IMO.

@frejen

Didn't mean to jack your tread.  Hope your question was answered... either way (help about_logical_operator)... Doesn't matter what I'm working on. I always have an extra posh console up just to refer back to the help docs.


~Glenn

bsonposhUser is Offline
CLI Addict
CLI Addict
Posts:363

07/24/2008 5:11 AM  
Again, it is a problem of scale. How big is the return set? How big are the DCs hardware wise? How much memory does the client have? I can go on and on.

In this case it is also a mater of simplicity and power. I can use a Regex client side... I cannot do that with a filter.

Don't get me wrong... I am a huge fan of ldap Filters and have tried my darndess to spread that love ( just check my blog :) )

Brandon Shell
----------------
Microsoft Powershell MVP
https://mvp.support.microsoft.com/profile/Brandon
frejenUser is Offline
New Member
New Member
Posts:8

07/24/2008 7:26 AM  
Thank you all for your replies.

Bsonposh > I want the result to be user objects where no one has SA: or SC: in their Desc. So i do NOT want users that match SA: or SC: to be returned. Therefore i used -inotmatch -or -inotmatch.

I guessed that would give me the users that does not have SA: or SC: in their description.
bsonposhUser is Offline
CLI Addict
CLI Addict
Posts:363

07/24/2008 9:15 AM  
@frejen, So your clear then?

Brandon Shell
----------------
Microsoft Powershell MVP
https://mvp.support.microsoft.com/profile/Brandon
frejenUser is Offline
New Member
New Member
Posts:8

07/25/2008 2:01 AM  
Yes iam using the LDAP filter and it seems to behave exactly as i want. Performance is not a concern this time.
But do i have the right picture of the -inotmatch and -or operators:
"I do NOT want users that match SA: or SC: to be returned. Therefore i used -inotmatch -or -inotmatch."
bsonposhUser is Offline
CLI Addict
CLI Addict
Posts:363

07/25/2008 5:22 AM  
I am not sure you do... Perhaps this will help. When you work with contiditional statements it is important to understand that it evaluates each conditional blcok by its self, and then steps out from there.

Look at this example

$myDescriptions = 1..2 | %{"Really descriptive 's$_' description"}
$myDescriptions += "My Description has an 'SC:' in it"
$myDescriptions += "My Description has an 'SA:' in it"

foreach($description in $myDescriptions)
{
    $matchSC                = $description  -notmatch "SC:"
    $matchSA                = $description  -notmatch "SA:"
    $UsingOr                = ($description -notmatch "SC:") -or ($description -notmatch "SA:")
    $UsingNotMatchCorrectly = $description  -notmatch "(SC|SA):"
    
    "Value:                    $description"
    "Match SC:                 $matchSC"
    "Match SA:                 $matchSA"
    "The Or:                   $UsingOr"  
    "Using NotMatch Correctly: $UsingNotMatchCorrectly"
    
    Write-Host
}

###############################################################
Value:                    Really descriptive 's1' description
Match SC:                 True
Match SA:                 True
The Or:                   True
Using NotMatch Correctly: True

Value:                    Really descriptive 's2' description
Match SC:                 True
Match SA:                 True
The Or:                   True
Using NotMatch Correctly: True

Value:                    My Description has an 'SC:' in it
Match SC:                 False
Match SA:                 True
The Or:                   True
Using NotMatch Correctly: False

Value:                    My Description has an 'SA:' in it
Match SC:                 True
Match SA:                 False
The Or:                   True
Using NotMatch Correctly: False
###############################################################

Brandon Shell
----------------
Microsoft Powershell MVP
https://mvp.support.microsoft.com/profile/Brandon
You are not authorized to post a reply.
Forums > Using PowerShell > General PowerShell > Problem with where and -or



ActiveForums 3.7
right
   
footer Sponsored by Quest Software • SAPIEN Technologies • ShellTools, LLC • Microsoft Windows Server 2008 footer
footer